Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRI
Not on the map. Why?
Abstract (arXiv)
The reconstruction of images observed by subjects from fMRI data collected during visual stimuli has made strong progress in the past decade, thanks to the availability of extensive fMRI datasets and advancements in generative models for image generation. However, the application of visual reconstruction has remained limited. Reconstructing visual imagination presents a greater challenge, with potentially revolutionary applications ranging from aiding individuals with disabilities to verifying witness accounts in court. The primary hurdles in this field are the absence of data collection protocols for visual imagery and the lack of datasets on the subject. Traditionally, fMRI-to-image relies on data collected from subjects exposed to visual stimuli, which poses issues for generating visual imagery based on the difference of brain activity between visual stimulation and visual imagery. For the first time, we have compiled a substantial dataset (around 6h of scans) on visual imagery along with a proposed data collection protocol. We then train a modified version of an fMRI-to-image model and demonstrate the feasibility of reconstructing images from two modes of imagination: from memory and from pure imagination. The resulting pipeline we call Mind-to-Image marks a step towards creating a technology that allow direct reconstruction of visual imagery.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 8 Apr 2024.
Builds on
14 of 24 references.
References · 14
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
- 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
- 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
- 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
- 2014Machine Learning for Neuroimaging with Scikit-Learncited by 70
- 2023Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activitycited by 69
Built on by
Citing papers per year: 2025: 2.
2 papers.