mlsanityData to 7 Oct 2026

From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

arXiv 1907.02431 ↗First posted 3 Jul 2019eess.IV cs.LG q-bio.NCcited by 67

Not on the map. Why?

Abstract (arXiv)

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 17; 2024: 18; 2025: 13; 2026: 7.

67 papers.

Sort:

By area:

  1. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
  2. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  3. 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
  4. 2022Mind Reader: Reconstructing complex images from brain activitiescited by 85
  5. 2023Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activitycited by 69
  6. 2024MindBridge: A Cross-Subject Brain Decoding Frameworkcited by 65
  7. 2022Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANscited by 55
  8. 2022Decoding natural image stimuli from fMRI data with a surface-based convolutional networkcited by 48
  9. 2023MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusioncited by 47
  10. 2023UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activitycited by 42
  11. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  12. 2023BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstructioncited by 30
38 more
  1. 2023Controllable Mind Visual Diffusion Modelcited by 29
  2. 2025Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Priorcited by 28
  3. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  4. 2024Neuro-3D: Towards 3D Visual Decoding from EEG Signalscited by 23
  5. 2024Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activitycited by 22
  6. 2023Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)cited by 20
  7. 2022A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activitycited by 19
  8. 2023Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signalscited by 16
  9. 2023fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decodingcited by 16
  10. 2023Brain-Conditional Multimodal Synthesis: A Survey and Taxonomycited by 15
  11. 2024The Wisdom of a Crowd of Brains: A Universal Brain Encodercited by 15
  12. 2022Visio-Linguistic Brain Encodingcited by 13
  13. 2024NeuroCine: Decoding Vivid Video Sequences from Human Brain Activtiescited by 10
  14. 2024Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activitycited by 10
  15. 2025BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activitiescited by 10
  16. 2021Natural Image Reconstruction from fMRI using Deep Learning: A Surveycited by 9
  17. 2023Joint fMRI Decoding and Encoding with Latent Embedding Alignmentcited by 8
  18. 2024Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understandingcited by 7
  19. 2023MindGPT: Interpreting What You See with Non-invasive Brain Recordingscited by 6
  20. 2024Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Modelscited by 6
  21. 2025A Survey of fMRI to Image Reconstructioncited by 6
  22. 2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
  23. 2021More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activitycited by 5
  24. 2021Neural Photofit: Gaze-based Mental Image Reconstructioncited by 5
  25. 2024See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRIcited by 5
  26. 2024MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decodingcited by 4
  27. 2025Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformercited by 4
  28. 2022Multi-view Multi-label Fine-grained Emotion Decoding from Human Brain Activitycited by 3
  29. 2023DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embeddingcited by 3
  30. 2024BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Modelscited by 3
  31. 2025fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignmentcited by 3
  32. 2024MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Modelcited by 2
  33. 2025Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representationscited by 2
  34. 2025From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structurecited by 2
  35. 2022Cross-view Brain Decodingcited by 1
  36. 2022Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activitycited by 1
  37. 2024Deep Learning for Visual Neuroprosthesiscited by 1
  38. 2024Scaling up ridge regression for brain encoding in a massive individual fMRI datasetcited by 1

50 most cited of 67 All 67 (JSON)