Neuroscientists have trained a neural network to translate brain signals into intelligible speech

Using the technology to scan brain activity, artificial intelligence and speech synthesis, scientists from Columbia University (USA) have created a device that can translate a person’s thoughts in articulate speech. The results of a study published in the journal Scientific Reports, represent an important step in improving interfaces “brain — computer”. In the future, such devices will be able to use the people who have lost the ability to speak as a result of injury or disease.

To develop the device that connects the possibilities of speech synthesis and artificial intelligence the study’s author, neuroscientist Nimes, Masharani and his colleagues turned to the most recent advances in deep machine learning and technology speech synthesis. The result of their work was the vocoder on the basis of artificial intelligence, the ability to accurately interpret brain activity directly from the auditory cortex of the brain, and then translate it into visible speech. The authors note that the speech in this case is highly computerized, but people can recognize words in most cases.

According to the creators, the new device used for playback of the resulting speech, used the same technology that is used in digital assistants such as the Alexa, Siri and Google Assistant.

The team first trained the vocoder correctly interpret the brain activity of a person. To do this, scientists were invited to participate in the experiment, five volunteers, who at the time was undergoing treatment for epilepsy. All five in the auditory cortex was implanted electrodes to read the electroencephalogram.

“We asked patients with epilepsy who are undergoing surgical treatment on the brain, listen to suggestions, which is pronounced by different people. At the same time, we analyzed patterns in the brain activity of patients. The obtained neural models and studied the vocoder,” says Massarani.

The patients were asked to listen to the recording, in which actors read out sequences of digits from 0 to 9. At the same time, the scientists recorded brain signals, and then passed it through a vocoder, the signals for which to enhance the clarity adjust the neural network, analyze the sounds of the vocoder, uttering sounds in response to these signals. As a result, you could hear a robotic voice, repeating the sequence of spoken numbers. To measure results, researchers recruited 11 people with excellent hearing.

“It turned out that people can recognize the words about 75% of the cases, significantly superior to any previous attempts. Delicate vocoder and powerful neural network generated sounds that are listened to patients with surprising accuracy”, — says Massarani.

In the future, the team Masharani’s going to teach a neural network to pronounce more complex words, phrases and even whole sentences. After that, they want to develop an implant that can translate human thoughts into a full-fledged speech.

“For example, if the owner of the implant will think: “I need a glass of water” — our system considers the brain signals and translate them to speech. This will give anyone who has lost the ability to speak due to injury or illness, a new opportunity to communicate with the outside world,” adds Masharani.

To discuss the development of the American scientists in our Telegram chat.

Leave a Reply

Your email address will not be published. Required fields are marked *