# Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks

arXiv: https://arxiv.org/abs/1810.03856 · HTML: https://mlsanity.com/paper/1810.03856/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2018.

## Built on by (most cited first, 30 of 43)

- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [MindBridge: A Cross-Subject Brain Decoding Framework](https://mlsanity.com/paper/2404.07850/) (2024)
- [Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs](https://mlsanity.com/paper/2202.12692/) (2022)
- [Decoding natural image stimuli from fMRI data with a surface-based convolutional network](https://mlsanity.com/paper/2212.02409/) (2022)
- [Brain Captioning: Decoding human brain activity into images and text](https://mlsanity.com/paper/2305.11560/) (2023)
- [A Systematic Collection of Medical Image Datasets for Deep Learning](https://mlsanity.com/paper/2106.12864/) (2021)
- [An OpenMind for 3D medical vision self-supervised learning](https://mlsanity.com/paper/2412.17041/) (2024)
- [More Real than Real: A Study on Human Visual Perception of Synthetic Faces](https://mlsanity.com/paper/2106.07226/) (2021)
- [BolT: Fused Window Transformers for fMRI Time Series Analysis](https://mlsanity.com/paper/2205.11578/) (2022)
- [NeuroGen: activation optimized image synthesis for discovery neuroscience](https://mlsanity.com/paper/2105.07140/) (2021)
- [Scaling laws for decoding images from brain activity](https://mlsanity.com/paper/2501.15322/) (2025)
- [Through their eyes: multi-subject Brain Decoding with simple alignment techniques](https://mlsanity.com/paper/2309.00627/) (2023)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [DreaMR: Diffusion-driven Counterfactual Explanation for Functional MRI](https://mlsanity.com/paper/2307.09547/) (2023)
- [Personalized visual encoding model construction with small data](https://mlsanity.com/paper/2202.02245/) (2022)
- [Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli](https://mlsanity.com/paper/2212.06726/) (2022)
- [Natural Image Reconstruction from fMRI using Deep Learning: A Survey](https://mlsanity.com/paper/2110.09006/) (2021)
- [A large-scale heterogeneous 3D magnetic resonance brain imaging dataset for self-supervised learning](https://mlsanity.com/paper/2506.14432/) (2025)
- [Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understanding](https://mlsanity.com/paper/2411.13378/) (2024)
- [A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli](https://mlsanity.com/paper/2503.15978/) (2025)
- [SPD Matrix Learning for Neuroimaging Analysis: Perspectives, Methods, and Challenges](https://mlsanity.com/paper/2504.18882/) (2025)
- [Compressive Ptychography using Deep Image and Generative Priors](https://mlsanity.com/paper/2205.02397/) (2022)
- [MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding](https://mlsanity.com/paper/2405.17720/) (2024)
- [SPI-GAN: Towards Single-Pixel Imaging through Generative Adversarial Network](https://mlsanity.com/paper/2107.01330/) (2021)
- [DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embedding](https://mlsanity.com/paper/2306.10082/) (2023)
- [Comparing interpretation methods in mental state decoding analyses with deep learning models](https://mlsanity.com/paper/2205.15581/) (2022)
- [Modality-Agnostic fMRI Decoding of Vision and Language](https://mlsanity.com/paper/2403.11771/) (2024)

## Abstract (arXiv)

While objects from different categories can be reliably decoded from fMRI brain response patterns, it has proved more difficult to distinguish visually similar inputs, such as different instances of the same category. Here, we apply a recently developed deep learning system to the reconstruction of face images from human fMRI patterns. We trained a variational auto-encoder (VAE) neural network using a GAN (Generative Adversarial Network) unsupervised training procedure over a large dataset of celebrity faces. The auto-encoder latent space provides a meaningful, topologically organized 1024-dimensional description of each image. We then presented several thousand face images to human subjects, and learned a simple linear mapping between the multi-voxel fMRI activation patterns and the 1024 latent dimensions. Finally, we applied this mapping to novel test images, turning the obtained fMRI patterns into VAE latent codes, and ultimately the codes into face reconstructions. Qualitative and quantitative evaluation of the reconstructions revealed robust pairwise decoding (>95% correct), and a strong improvement relative to a baseline model (PCA decomposition). Furthermore, this brain decoding model can readily be recycled to probe human face perception along many dimensions of interest; for example, the technique allowed for accurate gender classification, and even to decode which face was imagined, rather than seen by the subject. We hypothesize that the latent space of modern deep learning generative models could serve as a valid approximation for human brain representations.