Patients with paralysis can speak and move via a virtual avatar when using a new device created by scientists.
asserted
Patients → speak → scientists
The new “brain computer interface” (BCI) is the first which enables speech and movement activities in one device.
asserted
which → enable → device
Previous devices have helped paralysed patients speak or demonstrate movement, but not simultaneously.
asserted
patients → help → movement
The device allows brain activity be recorded in real time enabling the patient to control a personalised virtual avatar which enables gesture and communication.
asserted
which → allow → gesture
The research team, led by experts from the University of California, San Francisco, said that natural communication often combines words and movement including facial expressions and manual gestures.
asserted
communication → lead → expressions
Yet previous work on BCIs, which work by detecting brain activity in the sensorimotor cortex of the brain, have looked at speech and movement in isolation.
asserted
which → work → isolation
Until now, it has been “unclear whether a single BCI implant can reliably decode speech and gestures when they are attempted together”, the authors said.
uncertain
authors → decode → speech
Researchers set out to see whether a BCI used to decode brain signals to make an avatar replicate face and arm movements in three patients with severe paralysis.
asserted
avatar → set → paralysis
First they tested the device which translates brain activity and “decodes” it to create commands sent to external device.
asserted
which → test → device
They wanted to test the feasibility of real time speech and gesture “decoders” to see whether they could drive a personalised full-body avatar.
uncertain
they → want → avatar
First they assessed the use of the BCI to decode movement and speech separately using a set of phrases and gestures, and then both together.
asserted
they → assess → phrases
The team then assessed whether two patients could control a personalised virtual avatar with speech and movement simultaneously, including shrugging, fist pumps or nodding the head.
uncertain
patients → assess → head
One of the patients 100% accuracy for both speech and gesture decoding in three of the assessments, the research team found.
asserted
team → find → assessments
The brain implant could benefit people paralysed by stroke or neurogenerative diseases such as motor neurone disease, which can impair both speech and movement, reducing a person’s ability to interact with others, they said.
uncertain
they → benefit → others
But they said that testing in more participants and with larger sets of words and gestures will be needed before the technology can be used by patients.
asserted
technology → say → patients
This simultaneous, or “multi-effector”, speech and movement could help patients with speech and paralysis have “expressive interaction”, the authors said.
uncertain
authors → help → interaction
“These findings support the feasibility of simultaneous, multi-effector decoding from a single implant as a step toward more flexible, expressive communication for people with severe bodily and vocal-tract paralysis,” they wrote in the journal Nature Neuroscience.
asserted
they → support → journal
Researcher Samantha Brosler, PhD candidate in bioengineering at UCSF–UC Berkeley, told the Press Association: “We recorded brain activity from people with paralysis using electrodes implanted on the surface of the brain while they attempted to speak, make gestures, or do both at the same time.
asserted
they → tell → time
“We then trained machine learning models to translate brain activity into attempted phrases or body movements.
asserted
We → train → phrases
“We also created a personalised, full-body avatar for each participant using the video game development platform Unreal Engine.
asserted
We → create → platform
“In real time, we ran a speech decoder and a gesture decoder in parallel.
asserted
we → run → parallel
“If the participant attempted speech, the decoded phrase appeared as text on the screen; if they attempted a gesture, the avatar performed the corresponding movement.
asserted
avatar → attempt → movement
Because the two decoders operated independently, participants could use speech alone, a gesture alone, or both at the same time.”
uncertain
participants → operate → time
She added: “The participants were excited to see the system working in real time.
“One participant even started laughing when he saw that his speech and gestures were being successfully decoded together and acted out through the avatar.
asserted
speech → add → avatar
“The participants were also involved in designing how they wanted their personalised virtual avatars to look, so they were invested in the process from the beginning.
asserted
they → involve → beginning
“I think that made it all the more meaningful for them to ultimately control those avatars themselves using their own brain activity.”
asserted
them → think → activity
She went on: “To our knowledge, this is the first demonstration that speech and communicative gestures can be decoded at the same time from a single brain implant in people with paralysis, as well as the first demonstration of BCI-enabled control of a full-body virtual avatar.
asserted
speech → go → avatar
“This is still an early proof of concept, with simultaneous speech and gesture decoding tested in two participants using a constrained set of phrases and gestures.
asserted
decoding → test → phrases
“But it shows the potential for a single brain implant to combine multiple forms of communication, including speech and body movements, to support more flexible and expressive interaction for people with paralysis.”
asserted
implant → show → paralysis
Commenting on the research, Professor Deb Lowe, medical director for the Stroke Association said: “Every day, around 240 people in the UK survive a stroke.
asserted
people → comment → stroke
“For many, the resulting brain injury can have profound and lasting impacts, leaving stroke survivors with disabilities such as difficulty walking or talking.
asserted
injury → result → difficulty
“Specialist stroke rehabilitation will always be essential to help stroke survivors regain independence and some evidence-based technological innovations could complement existing support.
uncertain
innovations → help → support
“So, we welcome such research which could provide stroke survivors with the opportunity to communicate more effectively.
uncertain
which → welcome → opportunity
“However, this research is in its early clinical stages, and further exploration is required to fully understand how it could truly benefit stroke survivors.”
uncertain
it → require → survivors
