Data to 7 Oct 2026

MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

arXiv 2403.11207 ↗First posted 17 Mar 2024cs.CV cs.AI q-bio.NCcited by 105

Not on the map. Why?

Abstract (arXiv)

Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on GitHub.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 49 of its 97 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

49 of 97 references.

References · 49

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
2014Microsoft COCO: Common Objects in Contextcited by 17,498
2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903

37 more

2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
2023Getting aligned on representational alignmentcited by 142
2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
2022Mind Reader: Reconstructing complex images from brain activitiescited by 85
2019From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRIcited by 67
2023Brain decoding: toward real-time reconstruction of visual perceptioncited by 64
2020MixCo: Mix-up Contrastive Learning for Visual Representationcited by 62
2022Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANscited by 55
2022Decoding natural image stimuli from fMRI data with a surface-based convolutional networkcited by 48
2023DREAM: Visual Decoding from Reversing Human Visual Systemcited by 44
2023UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activitycited by 42
2023Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activitiescited by 36
2023Brain Captioning: Decoding human brain activity into images and textcited by 33
2023Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Modelscited by 29
2022Aligning individual brains with Fused Unbalanced Gromov-Wassersteincited by 23
2023Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputscited by 23
2023Through their eyes: multi-subject Brain Decoding with simple alignment techniquescited by 21
2023BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivitycited by 19
2019DCNN-GAN: Reconstructing Realistic Image from fMRIcited by 16
2023Aligning brain functions boosts the decoding of visual semantics in novel subjectscited by 9
2021Learning shared neural manifolds from multi-subject FMRI datacited by 8
2023Reconstructing seen images from human brain activity via guided stochastic searchcited by 8
2023Brain-optimized inference improves reconstructions of fMRI brain activitycited by 8
2023Second Sight: Using brain-optimized encoding models to align image distributions with human brain activitycited by 7
2023Memory Encoding Modelcited by 7
2023A Parameter-efficient Multi-subject Model for Predicting fMRI Activitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 48; 2026: 40.

105 papers.

Sort:

By area:

2024NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulationcited by 33
2024MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correctioncited by 31
2025Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Priorcited by 28
2024NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstructioncited by 24
2024Neuro-3D: Towards 3D Visual Decoding from EEG Signalscited by 23
2025Scaling laws for decoding images from brain activitycited by 21
2024The Wisdom of a Crowd of Brains: A Universal Brain Encodercited by 15
2025MindSimulator: Exploring Brain Concept Localization via Synthetic FMRIcited by 15
2025Towards a general-purpose foundation model for fMRI analysiscited by 15
2025MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decodingcited by 14
2024Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformerscited by 13
2025NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignmentcited by 13

38 more

2024Mind's Eye: Image Recognition by EEG via Multimodal Similarity-Keeping Contrastive Learningcited by 12
2026A foundation model of vision, audition, and language for in-silico neurosciencecited by 12
2025CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processingcited by 11
2025BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activitiescited by 10
2025SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learningcited by 9
2024Toward Generalizing Visual Brain Decoding to Unseen Subjectscited by 7
2024Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understandingcited by 7
2025Human-Aligned Image Models Improve Visual Decoding from the Braincited by 7
2025MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decodingcited by 7
2025NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagerycited by 7
2025A Survey of fMRI to Image Reconstructioncited by 6
2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
2025ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decodingcited by 6
2026FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRIcited by 6
2024Reanimating Images using Neural Representations of Dynamic Stimulicited by 5
2025Neurons: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstructioncited by 5
2025Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRIcited by 5
2025Scaling Vision Transformers for Functional MRI with Flat Mapscited by 5
2025BrainExplore: Large-Scale Discovery of Interpretable Visual Representations in the Human Braincited by 5
2024COBRA: A Continual Learning Approach to Vision-Brain Understandingcited by 4
2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
2025Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortexcited by 4
2025Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)cited by 4
2025Neural-Driven Image Editingcited by 4
2025Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformercited by 4
2025A cross-species neural foundation model for end-to-end speech decodingcited by 4
2026HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignmentcited by 4
2024BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Modelscited by 3
2025Neuroprobe: Evaluating Intracranial Brain Responses to Naturalistic Stimulicited by 3
2024Mind-to-Image: Projecting Visual Mental Imagination of the Brain from fMRIcited by 2
2024Alljoined1 -- A dataset for EEG-to-Image decodingcited by 2
2025TROI: Cross-Subject Pretraining with Sparse Voxel Selection for Enhanced fMRI Visual Decodingcited by 2
2025Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processingcited by 2
2025Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representationscited by 2
2025Multigranular Evaluation for Brain Visual Decodingcited by 2
2025A Cognitive Process-Inspired Architecture for Subject-Agnostic Brain Visual Decodingcited by 2
2026CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignmentcited by 2
2026Subject-Aware Multi-Granularity Alignment for Zero-Shot EEG-to-Image Retrievalcited by 2

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way