World first: brain implant lets man speak with expression and sing] (Source: Source: The study of a 45-year-old man with severe speech disability)
A man with a severe speech disability is able to speak expressively and sing using a brain implant that translates his neural activity into words almost instantly. The device conveys changes of tone when he asks questions, emphasizes the words of his choice and allows him to hum a string of notes in three pitches.
Christian Herff, a computational neuroscience professor at the Maastricht University in the Netherlands, doesn’t think that this is the panacea for speech BCIs. This is a continuous speech.
The study participant, a 45-year-old man, lost his ability to speak clearly after developing amyotrophic lateral sclerosis, a form of motor neuron disease, which damages the nerves that control muscle movements, including those needed for speech. Although he could still make sounds and mouth words, his speech was slow and unclear.
Source: World first: brain implant lets man speak with expression ― and sing
Neural Activity and Biosignal Intelligibility in a Clinical Trial of Synthesis of Speech using a Stereotactic Mic Decoder
Words do not always communicate what we want. We have interjections. We have other expressive vocalizations that are not in the vocabulary,” explains Wairagkar. “In order to do that, we have adopted this approach, which is completely unrestricted.”
By training artificial intelligence on recordings of interviews that the man had done before his disease, the team personalizes the synthetic voice to sound like him.
a. Three example trials’ audio recording, audio spectrogram, and the spectrograms of the two most acoustic-correlated neural electrodes. The three types of speech tasks are shown in examples. The prominent spectral structures in the audio spectrogram cannot be observed even in the top two most correlated neural electrodes. There is an increase in neural activity before each word’s start and there is more arguing against acoustic contamination. Note that in the word emphasis example, the last word ‘going’ is not vocalized fully (there is minimal activity in its audio spectrum), yet an increase in neural activity can be observed that is similar to other words. Contamination matrices and statistical criteria are shown in the bottom row, where P-value indicates whether the trial is significantly acoustically contaminated or not. b. A trial of a speech with simultaneous recording of neural signals and biosignals using a microphone, IMU sensors and stethoscope. The speech was synthesised using the biosignals or all three together. c. Speech could not be created using biosignals measuring movement, and sound. The Pearson correlation coefficients can be cross-validated. Speech is synthesized using Neural signals, each of the biosignals, and all biosignals together. Neural activity is more accurate for decoding speech from biosignals than for reconstruction. Pearson correlation coefficients of speech decoding from biosignals are mostly non-overlapping, suggesting that synthesis quality is lower than that of neural signals. d. To assess the intelligibility of voice synthesis from neural activity and biosignals (stethoscopic mic decoder), naive human listeners performed open transcription of (the same) 30 synthesized trials using both the decoders. Median phoneme error rates and word error rates for neural decoding were significantly lower (43.60%) than decoding stethoscope recordings, which had word error rate of 100%. There is no way intelligible speech can be taken from these biosignals.
