# From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

arXiv: https://arxiv.org/abs/1907.02431 · HTML: https://mlsanity.com/paper/1907.02431/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2019.

## Built on by (most cited first, 30 of 67)

- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [MindBridge: A Cross-Subject Brain Decoding Framework](https://mlsanity.com/paper/2404.07850/) (2024)
- [Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs](https://mlsanity.com/paper/2202.12692/) (2022)
- [Decoding natural image stimuli from fMRI data with a surface-based convolutional network](https://mlsanity.com/paper/2212.02409/) (2022)
- [MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion](https://mlsanity.com/paper/2308.04249/) (2023)
- [UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity](https://mlsanity.com/paper/2308.07428/) (2023)
- [BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding](https://mlsanity.com/paper/2302.12971/) (2023)
- [Controllable Mind Visual Diffusion Model](https://mlsanity.com/paper/2305.10135/) (2023)
- [BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction](https://mlsanity.com/paper/2312.14871/) (2023)
- [Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior](https://mlsanity.com/paper/2503.04207/) (2025)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [Neuro-3D: Towards 3D Visual Decoding from EEG Signals](https://mlsanity.com/paper/2411.12248/) (2024)
- [Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity](https://mlsanity.com/paper/2403.20022/) (2024)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity](https://mlsanity.com/paper/2206.03544/) (2022)
- [fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding](https://mlsanity.com/paper/2311.00342/) (2023)
- [Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals](https://mlsanity.com/paper/2308.02510/) (2023)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [The Wisdom of a Crowd of Brains: A Universal Brain Encoder](https://mlsanity.com/paper/2406.12179/) (2024)
- [Visio-Linguistic Brain Encoding](https://mlsanity.com/paper/2204.08261/) (2022)
- [Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity](https://mlsanity.com/paper/2405.03280/) (2024)
- [NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties](https://mlsanity.com/paper/2402.01590/) (2024)
- [BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activities](https://mlsanity.com/paper/2501.14309/) (2025)
- [Natural Image Reconstruction from fMRI using Deep Learning: A Survey](https://mlsanity.com/paper/2110.09006/) (2021)
- [Joint fMRI Decoding and Encoding with Latent Embedding Alignment](https://mlsanity.com/paper/2303.14730/) (2023)
- [Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understanding](https://mlsanity.com/paper/2411.13378/) (2024)

## Abstract (arXiv)

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.