mlsanityData to 7 Oct 2026

Decoding natural image stimuli from fMRI data with a surface-based convolutional network

arXiv 2212.02409 ↗First posted 5 Dec 2022cs.CV cs.LG q-bio.QMcited by 48

Not on the map. Why?

Abstract (arXiv)

Due to the low signal-to-noise ratio and limited resolution of functional MRI data, and the high complexity of natural images, reconstructing a visual stimulus from human brain fMRI measurements is a challenging task. In this work, we propose a novel approach for this task, which we call Cortex2Image, to decode visual stimuli with high semantic fidelity and rich fine-grained detail. In particular, we train a surface-based convolutional network model that maps from brain response to semantic image features first (Cortex2Semantic). We then combine this model with a high-quality image generator (Instance-Conditioned GAN) to train another mapping from brain response to fine-grained image features using a variational approach (Cortex2Detail). Image reconstructions obtained by our proposed method achieve state-of-the-art semantic fidelity, while yielding good fine-grained similarity with the ground-truth stimulus. Our code is available at: https://github.com/zijin-gu/meshconv-decoding.git.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 13 of its 34 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

13 of 34 references.

References · 13

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2021Instance-Conditioned GANcited by 119
  6. 2019Spherical CNNs on Unstructured Gridscited by 76
  7. 2019From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRIcited by 67
  8. 2022Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANscited by 55
  9. 2018Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networkscited by 43
  10. 2021Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GANcited by 31
  11. 2021NeuroGen: activation optimized image synthesis for discovery neurosciencecited by 22
  12. 2019DCNN-GAN: Reconstructing Realistic Image from fMRIcited by 16
1 more
  1. 2017Sharing deep generative representation for perceived image reconstruction from human brain activitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 16; 2025: 12; 2026: 4.

48 papers.

Sort:

By area:

  1. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
  2. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  3. 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
  4. 2024MindBridge: A Cross-Subject Brain Decoding Frameworkcited by 65
  5. 2023MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusioncited by 47
  6. 2023DREAM: Visual Decoding from Reversing Human Visual Systemcited by 44
  7. 2023UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activitycited by 42
  8. 2024NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulationcited by 33
  9. 2024CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Informationcited by 26
  10. 2023Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputscited by 23
  11. 2024Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activitycited by 22
  12. 2023fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decodingcited by 16
36 more
  1. 2023Brain-Conditional Multimodal Synthesis: A Survey and Taxonomycited by 15
  2. 2024Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activitycited by 10
  3. 2025BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activitiescited by 10
  4. 2023Aligning brain functions boosts the decoding of visual semantics in novel subjectscited by 9
  5. 2023Reconstructing seen images from human brain activity via guided stochastic searchcited by 8
  6. 2023Brain-optimized inference improves reconstructions of fMRI brain activitycited by 8
  7. 2023Second Sight: Using brain-optimized encoding models to align image distributions with human brain activitycited by 7
  8. 2024Toward Generalizing Visual Brain Decoding to Unseen Subjectscited by 7
  9. 2024Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understandingcited by 7
  10. 2025NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagerycited by 7
  11. 2023Lite-Mind: Towards Efficient and Robust Brain Representation Networkcited by 6
  12. 2025A Survey of fMRI to Image Reconstructioncited by 6
  13. 2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
  14. 2024See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRIcited by 5
  15. 2024Reanimating Images using Neural Representations of Dynamic Stimulicited by 5
  16. 2025Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRIcited by 5
  17. 2024MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decodingcited by 4
  18. 2024COBRA: A Continual Learning Approach to Vision-Brain Understandingcited by 4
  19. 2023DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embeddingcited by 3
  20. 2024Brain-Streams: fMRI-to-Image Reconstruction with Multi-modal Guidancecited by 3
  21. 2024A Review of Latent Representation Models in Neuroimagingcited by 2
  22. 2025From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structurecited by 2
  23. 2023Retinotopy Inspired Brain Encoding Model and the All-for-One Training Recipecited by 1
  24. 2025Cross-Subject Mind Decoding from Inaccurate Representationscited by 1
  25. 2025Towards Interpretable Visual Decoding with Attention to Brain Representationscited by 1
  26. 2025HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusioncited by 1
  27. 2026SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstructioncited by 1
  28. 2023MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusioncited by 0
  29. 2024Looking through the mind's eye via multimodal encoder-decoder networkscited by 0
  30. 2024Optimized two-stage AI-based Neural Decoding for Enhanced Visual Stimulus Reconstruction from fMRI Datacited by 0
  31. 2025FreqSelect: Frequency-Aware fMRI-to-Image Reconstructioncited by 0
  32. 2025HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusioncited by 0
  33. 2025NeuroSwift: A Lightweight Cross-Subject Framework for fMRI Visual Reconstruction of Complex Scenescited by 0
  34. 2026SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidancecited by 0
  35. 2026Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decodingcited by 0
  36. 2026NeurIPS: Neuro-anatomical Inductive Priors for Sphere-based Brain Decodingcited by 0