mlsanityData to 7 Oct 2026

Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding

arXiv 2211.06956 ↗First posted 13 Nov 2022cs.CVcited by 142

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, reconstructing high-quality images with correct semantics from brain recordings is a challenging problem due to the complex underlying representations of brain signals and the scarcity of data annotations. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. We benchmarked our model qualitatively and quantitatively; the experimental results indicate that our method outperformed state-of-the-art in both semantic mapping (100-way semantic classification) and generation quality (FID) by 66% and 41% respectively. An exhaustive ablation study was also conducted to analyze our framework.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 33 of its 72 references that are in the map. First posted on arXiv 13 Nov 2022.

Builds on

33 of 72 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2013Auto-Encoding Variational Bayescited by 3,980
21 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  4. 2016Colorful Image Colorizationcited by 1,113
  5. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  6. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  7. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  8. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  9. 2015Generic decoding of seen and imagined objects using hierarchical visual featurescited by 119
  10. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  11. 2022Mind Reader: Reconstructing complex images from brain activitiescited by 85
  12. 2019From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRIcited by 67
  13. 2018BOLD5000: A public fMRI dataset of 5000 imagescited by 63
  14. 2022Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANscited by 55
  15. 2020Reconstructing Natural Scenes from fMRI Patterns using BigBiGANcited by 44
  16. 2022How to Understand Masked Autoencoderscited by 37
  17. 2021Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GANcited by 31
  18. 2018A Connectome Based Hexagonal Lattice Convolutional Network Model of the Drosophila Visual Systemcited by 4
  19. 2019Synaptic Delays for Temporal Feature Detection in Dynamic Neuromorphic Processorscited by 4
  20. 2019Decoding Neural Responses in Mouse Visual Cortex through a Deep Neural Networkcited by 3
  21. 2022The Brain-Inspired Decoder for Natural Visual Image Reconstructioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 33; 2024: 45; 2025: 34; 2026: 29.

142 papers.

Sort:

By area:

  1. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
  2. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  3. 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
  4. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  5. 2024MindBridge: A Cross-Subject Brain Decoding Frameworkcited by 65
  6. 2023DreamDiffusion: Generating High-Quality Images from Brain EEG Signalscited by 56
  7. 2023Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Modelscited by 53
  8. 2023MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusioncited by 47
  9. 2023DREAM: Visual Decoding from Reversing Human Visual Systemcited by 44
  10. 2023Brain Captioning: Decoding human brain activity into images and textcited by 33
  11. 2024NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulationcited by 33
  12. 2024MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correctioncited by 31
38 more
  1. 2023BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstructioncited by 30
  2. 2023Controllable Mind Visual Diffusion Modelcited by 29
  3. 2023Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Modelscited by 29
  4. 2024Masked Image Modeling: A Surveycited by 28
  5. 2025Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Priorcited by 28
  6. 2024NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstructioncited by 24
  7. 2023Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputscited by 23
  8. 2023UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human languagecited by 22
  9. 2024Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activitycited by 22
  10. 2023Through their eyes: multi-subject Brain Decoding with simple alignment techniquescited by 21
  11. 2025Scaling laws for decoding images from brain activitycited by 21
  12. 2023BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivitycited by 19
  13. 2024ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidancecited by 18
  14. 2023Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signalscited by 16
  15. 2023fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decodingcited by 16
  16. 2023Brain-Conditional Multimodal Synthesis: A Survey and Taxonomycited by 15
  17. 2025MindSimulator: Exploring Brain Concept Localization via Synthetic FMRIcited by 15
  18. 2025Towards a general-purpose foundation model for fMRI analysiscited by 15
  19. 2024The Silent Assistant: NoiseQuery as Implicit Guidance for Goal-Driven Image Generationcited by 14
  20. 2025MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decodingcited by 14
  21. 2024Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformerscited by 13
  22. 2022Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimulicited by 12
  23. 2024Latent Diffusion for Neural Spiking Datacited by 12
  24. 2023MinD-3D: Reconstruct High-quality 3D objects in Human Braincited by 11
  25. 2024Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Modelscited by 11
  26. 2024NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mappingcited by 11
  27. 2025CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processingcited by 11
  28. 2024Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activitycited by 10
  29. 2025BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activitiescited by 10
  30. 2023A Survey of Emerging Applications of Diffusion Probabilistic Models in MRIcited by 9
  31. 2023Aligning brain functions boosts the decoding of visual semantics in novel subjectscited by 9
  32. 2024Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversioncited by 9
  33. 2025SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learningcited by 9
  34. 2023Joint fMRI Decoding and Encoding with Latent Embedding Alignmentcited by 8
  35. 2024Hierarchical Quantum Control Gates for Functional MRI Understandingcited by 8
  36. 2023ChatTraffic: Text-to-Traffic Generation via Diffusion Modelcited by 7
  37. 2024Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understandingcited by 7
  38. 2025NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagerycited by 7

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Decoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion
  2. 2023Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
  3. 2023Reconstructing seen images from human brain activity via guided stochastic search
  4. 2024Brain-Streams: fMRI-to-Image Reconstruction with Multi-modal Guidance
  5. 2026Toward a mechanistic understanding of inference in visual cortex and diffusion models
  6. 2026Brain-Grasp: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding
  7. 2023Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models
  8. 2024NeuralDiffuser: Neuroscience-inspired Diffusion Guidance for fMRI Visual Reconstruction