Researchers decoded a paralyzed participant's neural activity in real time to control a personalized avatar, enabling communication through speech and gestures. Image credit: University of California, San Francisco

On the 14th, Nature Neuroscience published a latest neuroscience research finding: a single brain implant can simultaneously decode speech and gestures in paralyzed patients. A U.S. research team decoded a paralyzed participant's neural activity in real time to control a personalized avatar, enabling communication through speech and gestures. The results advance brain-computer interface technology and offer hope to help paralyzed patients communicate in a more natural way.

Natural human communication often combines speech with gestures, such as waving, nodding, or other upper-body movements. Stroke and neurodegenerative diseases, such as amyotrophic lateral sclerosis, can impair both speech and body movement, weakening patients' ability to socialize.

Brain-computer interfaces aim to translate brain activity into commands that control external devices, thereby restoring lost function. Brain-computer interface technology has accumulated over more than a decade. Earlier approaches mostly used cortical surface electrodes or microelectrodes penetrating the cortex to record neural activity in speech-related motor areas, then used machine learning to convert intended sounds, words, and sentences into text or synthesized speech; the motor direction long focused on arm and hand areas for cursor control, robotic arm control, or grasp control. However, speech and upper-limb/facial movements partially overlap in the sensorimotor cortex, and decoders trained separately for "speech only" or "movement only" may interfere with each other when run in parallel. Previous studies also mostly focused on speech or movement separately.

In this study, a team at the University of California, San Francisco used a brain implant placed above the sensorimotor cortex to record neural activity from three participants with vocal tract and limb paralysis. They first demonstrated that a single implant can capture signals related to multiple movement types, including upper-limb movements and speech-related mouth and facial movements, such as shrugging, fist swinging, or nodding; subsequently, the team used parallel speech and gesture decoders in two of the participants to drive a full-body avatar, and participants were asked to attempt specific gestures, respond to conversation prompts, or do both simultaneously.

In conversation tasks, the system accurately decoded the speech and gestures of two participants: one participant achieved a median accuracy of 100% for both speech and gesture decoding across three conversation task blocks.

The team found that training the system with both speech and gesture data improved performance and reduced errors. They noted that before this technology can be applied, it still needs to include more participants and be tested with larger vocabulary sets and gesture sets.