mlsanityData to 7 Oct 2026

DREAM: Visual Decoding from Reversing Human Visual System

arXiv 2310.02265 ↗First posted 3 Oct 2023cs.CV cs.LG eess.IVcited by 44

Not on the map. Why?

Abstract (arXiv)

In this work we present DREAM, an fMRI-to-image method for reconstructing viewed images from brain activities, grounded on fundamental knowledge of the human visual system. We craft reverse pathways that emulate the hierarchical and parallel nature of how humans perceive the visual world. These tailored pathways are specialized to decipher semantics, color, and depth cues from fMRI data, mirroring the forward pathways from visual stimuli to fMRI recordings. To do so, two components mimic the inverse processes within the human visual system: the Reverse Visual Association Cortex (R-VAC) which reverses pathways of this brain region, extracting semantics from fMRI data; the Reverse Parallel PKM (R-PKM) component simultaneously predicting color and depth from fMRI signals. The experiments indicate that our method outperforms the current state-of-the-art models in terms of the consistency of appearance, structure, and semantics. Code will be made publicly available to facilitate further research in this field.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

38 of 54 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
26 more
  1. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  2. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  3. 2015Holistically-Nested Edge Detectioncited by 590
  4. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  5. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  6. 2021Pixel Difference Networks for Efficient Edge Detectioncited by 167
  7. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
  8. 2022Balanced Contrastive Learning for Long-Tailed Visual Recognitioncited by 143
  9. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
  10. 2015Generic decoding of seen and imagined objects using hierarchical visual featurescited by 119
  11. 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
  12. 2022Mind Reader: Reconstructing complex images from brain activitiescited by 85
  13. 2020MixCo: Mix-up Contrastive Learning for Visual Representationcited by 62
  14. 2022Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANscited by 55
  15. 2021Self-Damaging Contrastive Learningcited by 48
  16. 2022Decoding natural image stimuli from fMRI data with a surface-based convolutional networkcited by 48
  17. 2023MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusioncited by 47
  18. 2020Reconstructing Natural Scenes from fMRI Patterns using BigBiGANcited by 44
  19. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  20. 2023Brain Captioning: Decoding human brain activity into images and textcited by 33
  21. 2023Controllable Mind Visual Diffusion Modelcited by 29
  22. 2023Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputscited by 23
  23. 2022PalGAN: Image Colorization with Palette Generative Adversarial Networkscited by 15
  24. 2021Semantic Palette: Guiding Scene Generation with Class Proportionscited by 14
  25. 2021More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activitycited by 5
  26. 2014Pixels to Voxels: Modeling Visual Representation in the Human Braincited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 23; 2026: 9.

44 papers.

Sort:

By area:

  1. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  2. 2024MindBridge: A Cross-Subject Brain Decoding Frameworkcited by 65
  3. 2024UMBRAE: Unified Multimodal Brain Decodingcited by 31
  4. 2025Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Priorcited by 28
  5. 2023Brain-Conditional Multimodal Synthesis: A Survey and Taxonomycited by 15
  6. 2025MindSimulator: Exploring Brain Concept Localization via Synthetic FMRIcited by 15
  7. 2024Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversioncited by 9
  8. 2024Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understandingcited by 7
  9. 2025NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagerycited by 7
  10. 2023Lite-Mind: Towards Efficient and Robust Brain Representation Networkcited by 6
  11. 2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
  12. 2024See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRIcited by 5
32 more
  1. 2024BrainBits: How Much of the Brain are Generative Reconstruction Methods Using?cited by 5
  2. 2025Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRIcited by 5
  3. 2024MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decodingcited by 4
  4. 2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
  5. 2025Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformercited by 4
  6. 2024Modality-Agnostic fMRI Decoding of Vision and Languagecited by 2
  7. 2024MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Modelcited by 2
  8. 2025Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processingcited by 2
  9. 2025Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representationscited by 2
  10. 2025Multigranular Evaluation for Brain Visual Decodingcited by 2
  11. 2025From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structurecited by 2
  12. 2026Autoregressive Visual Decoding from EEG Signalscited by 2
  13. 2026CognitionCapturerPro: Towards High-Fidelity Visual Decoding from EEG/MEG via Multi-modal Information and Asymmetric Alignmentcited by 2
  14. 2025RETRO: REthinking Tactile Representation Learning with Material PriOrscited by 1
  15. 2025Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEGcited by 1
  16. 2025Cross-Subject Mind Decoding from Inaccurate Representationscited by 1
  17. 2025Towards Interpretable Visual Decoding with Attention to Brain Representationscited by 1
  18. 2025HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusioncited by 1
  19. 2026SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstructioncited by 1
  20. 2025SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decodingcited by 0
  21. 2025Probability Distribution Alignment and Low-Rank Weight Decomposition for Source-Free Domain Adaptive Brain Decodingcited by 0
  22. 2025HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusioncited by 0
  23. 2025NeuroSwift: A Lightweight Cross-Subject Framework for fMRI Visual Reconstruction of Complex Scenescited by 0
  24. 2025Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRIcited by 0
  25. 2025Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstructioncited by 0
  26. 2025Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mappingcited by 0
  27. 2026Simple Models, Rich Representations: Visual Decoding from Primate Intracortical Neural Signalscited by 0
  28. 2026SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidancecited by 0
  29. 2026NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activitycited by 0
  30. 2026StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptationcited by 0
  31. 2026HyNeuralMap: Hyperbolic Mapping of Visual Semantics to Neural Hierarchiescited by 0
  32. 2026Gaze-to-text Generation: Beyond Categorical Decoding of Human Attentioncited by 0