A NIH-funded team built a brain-computer interface that lets paralyzed people speak and gesture at the same time. It uses sensors on the motor cortex to capture brain signals. Machine learning decoders turn those signals into commands for a full-body avatar. The system worked for two of three patients tested. Speech and gesture signals combined are different from either alone. Decoders trained on simultaneous data performed better. The current device is wired but a wireless version is planned. The work aims to restore natural, layered communication. ALS and brainstem stroke patients may benefit most. The study appears in Nature Neuroscience.
What changed
BCIs previously enabled either speech or gestures, not both at once.
What it unlocks
People with paralysis can now convey speech and upper-body gestures together through a digital avatar.
- 3 patients tested
- 2 participants controlled avatars
What you need to act on it
- implanted ECoG arrays
- machine learning decoders trained on simultaneous speech and gesture data
Sources