When we speak, words never travel alone. We move our hands, nod, shake our heads, and change our facial expressions. For a person with paralysis, losing movement can also mean losing much of that way of communicating.
A team at the University of California, San Francisco showed that a single brain-computer interface can record brain activity related to different movements and use it to decode attempted speech and gestures at the same time. In part of the experiment, those signals controlled a virtual avatar that displayed text and movements according to what the participant was trying to do.
The result, published in Nature Neuroscience, does not yet restore a complete conversation. But it demonstrates something important: one interface can begin to recover more than one form of expression at once.
Speaking has never been only about words
Imagine a conversation where you cannot raise an eyebrow, point at something, nod, or accompany a sentence with a gesture.
We could still exchange information, but something important would be missing.
Our bodies are part of how we speak.
That is the problem this study addresses. Brain-computer interfaces have already converted brain activity into text, moved cursors, or recognized attempted movements. But many of those systems have focused on one task at a time.
The researchers asked a different question:
Can a single implant tell that a person is trying to speak and gesture at the same time?
First: this is not mind reading
The system does not listen to an inner conversation or discover what someone is thinking.
Participants try to perform specific actions — for example, say a sentence or make a gesture — and the system learns to recognize patterns of brain activity associated with those attempts.
To record them, the researchers used electrocorticography, or ECoG: a technique in which an array of electrodes is surgically placed on the surface of the brain.
Those electrodes capture electrical changes related to the activity of nearby populations of neurons.
A written word does not simply appear inside the brain. What the system receives is a complex signal that changes over time.
That is where interpretation begins.
Where artificial intelligence comes in
Artificial intelligence is part of the system, but it is not the main story and it is not generative AI.
The researchers trained neural networks — machine-learning models that can detect patterns in large amounts of data — to turn brain signals into probabilities.
There was one decoder for speech and another for gestures. Both ran in parallel.
Simplified:
brain activity → neural network → probabilities → phrase, gesture, or rest
The models combined layers that detect local patterns in the signal with others designed to analyze how that signal changes over time.
That matters because trying to say a sentence does not produce one isolated electrical event. It produces a sequence.
The AI does not decide what a person wants to say. It learns to associate certain patterns of activity with actions that person is attempting to perform.
The brain does not simply add the two signals together
The study involved three people with severe paralysis. Two had experienced brainstem strokes and one had amyotrophic lateral sclerosis, or ALS.
First, the researchers showed that a single implant could capture information related to different movements of the face, mouth, hands, and other parts of the body.
The most important part of the simultaneous communication experiment was then studied in two participants.
And an interesting problem appeared.
When the models were trained only on attempts to speak alone or gesture alone, their performance dropped when participants were asked to do both at the same time.
The combined brain activity did not behave like a perfectly clean sum of two independent signals.
So the researchers did something fairly intuitive: they added training examples in which speech and gesture happened together.
Performance improved.
In other words, to recognize something closer to a real conversation, the system also had to learn from situations that looked more like a real conversation.
From a brain signal to an avatar
The researchers connected both decoders to a personalized virtual avatar.
When the system recognized an attempt to speak, it displayed the decoded phrase as text.
When it recognized a gesture, it animated the avatar's body.
And both could happen at the same time.
In one participant, identified as Bravo-6 in the study, the system worked with ten phrases and ten gestures. In a real-time task where it had to recognize both simultaneously, it reached about 70% accuracy for speech and 66% for gestures. Chance performance was around 9% for each decoder.
That does not mean the participant could have any conversation imaginable.
The vocabulary was limited and defined in advance.
The experiment asked a more specific question: whether one interface could handle two channels of expression in parallel.
The early answer was yes.
Why it matters
A communication technology can recover words and still leave out a huge part of human communication.
A "yes" accompanied by a smile does not communicate exactly the same thing as a "yes" accompanied by a doubtful gesture.
Pointing, nodding, shaking your head, waving, or moving your arms can add intention and context.
That is why this work is about more than increasing the speed at which someone can select letters on a screen.
It points toward recovering the ability to express yourself.
For someone who has lost much of the control of their body and voice, the difference between transmitting text alone and taking part in a conversation with words and gestures could be enormous.
What this study still does not show
This is where it is important not to get ahead of the science.
The study involved only three participants, and the specific speech-and-gesture experiments focused on two.
The vocabularies were small and defined in advance.
The system also requires a surgically implanted brain interface, and the experimental setup still depends on specialized infrastructure.
So this result does not mean that a device is available today for any person with paralysis to use in everyday life.
It also does not mean the system can interpret any word, gesture, or thought.
It is a proof of concept: evidence that the principle can work and is worth investigating further.
The authors point to exactly that next step — testing more participants, broader sets of speech and gestures, and systems that can work more flexibly beyond highly controlled tasks.
Recovering a conversation, not just a function
For a long time, an assistive technology could be framed around recovering one specific function: moving a cursor, selecting a letter, controlling a prosthetic device.
This experiment looks a little further.
It does not ask only how to help someone transmit words again.
It asks how to restore part of the richness with which people communicate.
Because communication is not only the transfer of information.
It is also the ability to point, react, emphasize, hesitate, greet someone, and express yourself in front of another person.
And if a brain-computer interface can one day restore some of that to someone who lost it, the advance will not exist only inside the implant.
It will exist in the conversation that becomes possible again.
