Can an AI decode speech from brain activity?

An artificial intelligence can decode words and sentences from brain activity with amazing, but still limited, accuracy. Using just seconds of brain activity data, the AI ​​guesses what a person has heard; lists the correct answer in its top 10 possibilities up to 73 percent of the time, researchers found in a preliminary study.

Developed at Facebook’s parent company Meta, the AI ​​could eventually be used to help thousands of people around the world who are unable to communicate via speech, keyboard, or gesture — that includes many patients in minimally conscious states, locked up or vegetative, what is now generally known as unresponsive wakefulness syndrome.

Most existing technologies to help such patients communicate require risky brain surgeries to implant electrodes, but this new approach could provide a viable way to help patients with communication deficits without the use of invasive methods, neuroscientists say. Researchers trained a computational tool to detect words and sentences in 56,000 hours of voice recordings from 53 languages; The tool, also known as a language model, learned to recognize specific language features both on a fine-grained level (think letters or syllables) and on a broader level, such as a word or a sentence.

The team applied an AI with this language model to databases from four institutions that included the brain activity of 169 volunteers. In these databases, participants listened to various stories and sentences from, for example, Ernest Hemingway’s The Old Man and the Sea and Lewis Carroll’s Alice’s Adventures in Wonderland while people’s brains were scanned using magnetoencephalography or electroencephalography, those techniques measure the magnetic or electrical component of brain signals.

Then, with the help of a computational method that helps explain the physical differences between real brains, the team tried to decode what the participants had heard, using just three seconds of brain activity data from each person. The team instructed the AI to align speech sounds from the recordings with patterns of brain activity that the AI calculated as corresponding to what people were hearing, then made predictions about what the person might have been hearing during that short time, given more than 1,000 possibilities.

Using magnetoencephalography, or MEG, the correct answer was among the AI’s 10 best guesses up to 73 percent of the time, the researchers found. With electroencephalography, that value was reduced to no more than 30 percent.