An AI system developed at the Weizmann Institute of Science can reconstruct images people are looking at from fMRI scans. The improvement is not just that it can identify a banana as a banana. It aims to recover details such as the image’s structure and the objects’ positions, while requiring much less calibration data from each new participant.
That is a meaningful research advance, not a consumer mind-reading device. The distinction matters, particularly as scientists begin exploring whether similar methods could work with EEG, which is easier to collect than an fMRI scan and brings a less comfortable set of privacy questions.
Two models, working together
The research team, led by computer scientist Michal Irani, trained its system using existing high-resolution fMRI data from eight people. Each had viewed around 9,000 images while in a scanner.
An fMRI scanner does not record thoughts or directly track individual neurons. It measures changes in oxygenated blood flow, which researchers use as an indicator of activity in different parts of the brain. In the datasets used by the team, each measured voxel covered about one cubic millimeter. In a typical scanner, a voxel may cover around three cubic millimeters. Even the smaller unit contains roughly 16,000 neurons, so the scan remains an indirect and relatively coarse view of what is happening.
The new system has two parts. One estimates the visual structure of an image, such as where colors and shapes appear. The other estimates its content, such as the presence of a plate of bananas. Their predictions guide a diffusion model, which produces a reconstructed image.
Earlier systems could sometimes infer the general object a person had seen, but might place it differently or fail to reproduce its structure. The goal here is a closer reconstruction, not merely a plausible image of the same category.
Training beyond the scans
Collecting fMRI data is slow and expensive, which limits the number of image-and-scan pairs available for training. Irani’s team addressed that constraint with another model, an encoder that predicts what a person’s brain scan would look like after seeing a particular image.
The researchers can feed an image into the encoder, use the predicted brain activity as input to the image decoder, and compare the resulting reconstruction with the original. Repeating this process helps train both models. It also lets the team use images that were never shown to a person in a scanner. About 70 percent of the training data came from images that were not originally paired with fMRI scans.
That is a useful change in the economics of the work. The system is not limited to the images for which researchers already have a matching scan, a dataset that is costly to expand by asking people to spend more time in a scanner.
The team also combined data from multiple studies to identify brain regions that appeared to respond to similar categories across participants. Some regions seemed to respond to food images, for example, while others responded to sports images. The researchers plan to work with neuroscientists to investigate what these patterns can reveal about the brain.
Less setup, imperfect results
For a new participant, the team’s universal brain encoder needs about one hour of fMRI data to make predictions about what that person is seeing. Earlier approaches typically required around 40 hours for a new person. That difference could make the method more practical for neuroscience studies, where scanning time is costly and a single hour has been estimated at $600 to $1,000.
The reconstructions are not consistently accurate. In examples described by Irani, the system turned a cake into something resembling three sandwiches, and reconstructed a dog in a bathtub as a goat in a bathtub. The results may preserve some visual qualities while getting the object itself wrong, which is a reminder that a convincing-looking reconstruction is not necessarily a faithful record of perception.
The team reports that its method performed substantially better than other systems in a comparison test, though no benchmark figures are provided in the report. The work was presented at the Cognitive Computational Neuroscience conference in New York.
Applications remain possible, not proven
One hoped-for use is communication for people who are locked in, meaning they are conscious but unable to move or speak. If brain activity could be translated into useful messages, the method might offer a new route to communication. That remains a prospect, rather than a capability demonstrated by the image reconstruction work described here.
Researchers are also interested in whether brain-decoding methods could help study PTSD flashbacks or eventually reconstruct dreams. Irani’s team plans to move beyond still images toward video and audio. The current system does not reconstruct dreams, and the researchers have not yet shown that it can decode the contents of imagined scenes in the way those future goals would require.
The privacy question gets more practical
For now, the approach depends on fMRI, which requires a person to lie still inside a large scanner. That makes covert use difficult. The system also depends on brain data, and its success with people viewing images should not be mistaken for proof that it can read any thought at will.
The concern is what may happen if related methods work with EEG. EEG measures electrical activity through electrodes, which can be worn in a cap or incorporated into devices such as headphones. Researchers are working on ways to decode brain activity with EEG, though the fMRI system described here is not itself an EEG product.
Neuroscientist and philosopher Marcello Ienca has raised the possibility that a company could extract additional information from a user’s brain activity after calibrating an EEG device to that person. He also sees a possible future role for reconstructed mental imagery in court, though that is a concern about what the technology might enable, not a current legal practice described in the research.
The nearer-term lesson is more modest. Better models and more efficient training are making brain-image reconstruction less demanding, while the current method still relies on expensive scans and produces visible mistakes. The science has moved ahead. The difficult question is how much privacy we should expect from devices designed to measure the brain, especially when their makers may eventually promise that they are only listening for something useful.