Is it possible
to fully reconstruct what someone sees based on brain signals alone? The answer
is no, not yet. But EPFL researchers have made an important step in that
direction by introducing a new algorithm for building artificial neural network
models that capture brain dynamics with an impressive degree of accuracy.
Rooted in mathematics, the novel machine learning
algorithm is called CEBRA (pronounced "zebra"), and learns the hidden
structure in the neural code.
The researchers' demos are clear. A mouse watches a
1960s, black and white movie clip of a man running to a car and opening the
trunk. On another screen, one sees a reconstruction of the movie as calculated
by CEBRA. The CEBRA-constructed movie almost matches the original completely,
with some slightly eerie distortions, as if you've just seen a bug in the
Matrix.
What information the CEBRA learns from the raw neural
data can be tested after training by decoding—a method that is used for
brain-machine-interfaces (BMIs)—and they've shown they can decode from the
model what a mouse sees while it watches a movie. But CEBRA is not limited to
visual cortex neurons, or even brain data. Their study also shows it can be
used to predict the movements of the arm in primates, and to reconstruct the
positions of rats as they freely run around an arena.
"This work is just one step towards the theoretically-backed algorithms that are needed in neurotechnology to enable high-performance BMIs," says Mackenzie Mathis, EPFL's Bertarelli Chair of Integrative Neuroscience and PI of the study, which has been published in Nature.
For learning
the latent (i.e., hidden) structure in the visual system of mice, CEBRA can
predict unseen movie frames directly from brain signals alone after an initial training period mapping brain signals and
movie features.
The data used for the video decoding was open-access
through the Allen Institute in Seattle, WA. The brain signals are obtained
either directly by measuring brain activity via electrode probes inserted into the visual cortex area of the
mouse's brain, or using optical probes which consist of using genetically
modified mice, engineered so that activated neurons glow green. During the
training period, CEBRA learns to map the brain activity to specific frames.
CEBRA performs well with less than 1% of neurons in the visual cortex,
considering that, in mice, this brain area consists of roughly 0.5 million
neurons.
"Concretely, CEBRA is based on contrastive
learning, a technique that learns how high-dimensional data can be arrange, or embedded, in a lower-dimensional space called a
latent space, so that similar data points are close together and more-different
data points are further apart," explains Mathis.
"This embedding can be used to infer hidden
relationships and structure in the data. It enables researchers to jointly
consider neural data and behavioral labels, including measured movements,
abstract labels like 'reward,' or sensory features such as colors or textures
of images."
"CEBRA excels compared to other algorithms at
reconstructing synthetic data, which is critical to compare algorithms,"
says Steffen Schneider, the co-first author of the paper. "Its strengths
also lie in its ability to combine data across modalities, such as movie
features and brain data, and it helps limit nuances, such as changes to the
data that depend on how they were collected."
"The goal of CEBRA is to uncover structure in complex systems. And, given the brain is the most complex structure in our universe,
it's the ultimate test space for CEBRA. It can also give us insight into how
the brain processes information and could be a platform for discovering new
principles in neuroscience by combining data across animals, and even
species," says Mathis.
"This algorithm is not limited to neuroscience research, as it can be applied to many datasets involving time or joint information, including animal behavior and gene-expression data. Thus, the potential clinical applications are exciting."
by Ecole
Polytechnique Federale de Lausanne
Source: Seeing through the eyes of a mouse by decoding its brain signals (medicalxpress.com)
No comments:
Post a Comment