# Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN

arXiv: https://arxiv.org/abs/2001.11761 · HTML: https://mlsanity.com/paper/2001.11761/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2020.

## Built on by (most cited first, 30 of 44)

- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://mlsanity.com/paper/2306.16934/) (2023)
- [Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs](https://mlsanity.com/paper/2202.12692/) (2022)
- [DREAM: Visual Decoding from Reversing Human Visual System](https://mlsanity.com/paper/2310.02265/) (2023)
- [UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity](https://mlsanity.com/paper/2308.07428/) (2023)
- [BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding](https://mlsanity.com/paper/2302.12971/) (2023)
- [Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities](https://mlsanity.com/paper/2305.17214/) (2023)
- [Brain Captioning: Decoding human brain activity into images and text](https://mlsanity.com/paper/2305.11560/) (2023)
- [NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation](https://mlsanity.com/paper/2403.18211/) (2024)
- [NeuroGen: activation optimized image synthesis for discovery neuroscience](https://mlsanity.com/paper/2105.07140/) (2021)
- [Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity](https://mlsanity.com/paper/2403.20022/) (2024)
- [fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding](https://mlsanity.com/paper/2311.00342/) (2023)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli](https://mlsanity.com/paper/2212.06726/) (2022)
- [NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties](https://mlsanity.com/paper/2402.01590/) (2024)
- [Natural Image Reconstruction from fMRI using Deep Learning: A Survey](https://mlsanity.com/paper/2110.09006/) (2021)
- [Joint fMRI Decoding and Encoding with Latent Embedding Alignment](https://mlsanity.com/paper/2303.14730/) (2023)
- [A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli](https://mlsanity.com/paper/2503.15978/) (2025)
- [ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding](https://mlsanity.com/paper/2510.27128/) (2025)
- [MindGPT: Interpreting What You See with Non-invasive Brain Recordings](https://mlsanity.com/paper/2309.15729/) (2023)
- [More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activity](https://mlsanity.com/paper/2106.05113/) (2021)
- [Neural Photofit: Gaze-based Mental Image Reconstruction](https://mlsanity.com/paper/2108.07524/) (2021)
- [Decoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion](https://mlsanity.com/paper/2310.00318/) (2023)
- [See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI](https://mlsanity.com/paper/2403.06361/) (2024)
- [Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer](https://mlsanity.com/paper/2510.25976/) (2025)
- [MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding](https://mlsanity.com/paper/2405.17720/) (2024)
- [DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embedding](https://mlsanity.com/paper/2306.10082/) (2023)
- [Understanding Cognitive Fatigue from fMRI Scans with Self-supervised Learning](https://mlsanity.com/paper/2106.15009/) (2021)

## Abstract (arXiv)

Decoding and reconstructing images from brain imaging data is a research area of high interest. Recent progress in deep generative neural networks has introduced new opportunities to tackle this problem. Here, we employ a recently proposed large-scale bi-directional generative adversarial network, called BigBiGAN, to decode and reconstruct natural scenes from fMRI patterns. BigBiGAN converts images into a 120-dimensional latent space which encodes class and attribute information together, and can also reconstruct images based on their latent vectors. We computed a linear mapping between fMRI data, acquired over images from 150 different categories of ImageNet, and their corresponding BigBiGAN latent vectors. Then, we applied this mapping to the fMRI activity patterns obtained from 50 new test images from 50 unseen categories in order to retrieve their latent vectors, and reconstruct the corresponding images. Pairwise image decoding from the predicted latent vectors was highly accurate (84%). Moreover, qualitative and quantitative assessments revealed that the resulting image reconstructions were visually plausible, successfully captured many attributes of the original images, and had high perceptual similarity with the original content. This method establishes a new state-of-the-art for fMRI-based natural image reconstruction, and can be flexibly updated to take into account any future improvements in generative models of natural scene images.