# Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN

arXiv: https://arxiv.org/abs/2101.12083 · HTML: https://mlsanity.com/paper/2101.12083/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2021.

## Built on by (most cited first, 30 of 32)

- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Decoding natural image stimuli from fMRI data with a surface-based convolutional network](https://mlsanity.com/paper/2212.02409/) (2022)
- [MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion](https://mlsanity.com/paper/2308.04249/) (2023)
- [UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity](https://mlsanity.com/paper/2308.07428/) (2023)
- [Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities](https://mlsanity.com/paper/2305.17214/) (2023)
- [BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction](https://mlsanity.com/paper/2312.14871/) (2023)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties](https://mlsanity.com/paper/2402.01590/) (2024)
- [Natural Image Reconstruction from fMRI using Deep Learning: A Survey](https://mlsanity.com/paper/2110.09006/) (2021)
- [A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli](https://mlsanity.com/paper/2503.15978/) (2025)
- [MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding](https://mlsanity.com/paper/2505.15946/) (2025)
- [MindGPT: Interpreting What You See with Non-invasive Brain Recordings](https://mlsanity.com/paper/2309.15729/) (2023)
- [A Survey of fMRI to Image Reconstruction](https://mlsanity.com/paper/2502.16861/) (2025)
- [Decoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion](https://mlsanity.com/paper/2310.00318/) (2023)
- [MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model](https://mlsanity.com/paper/2405.18812/) (2024)
- [Facial Image Reconstruction from Functional Magnetic Resonance Imaging via GAN Inversion with Improved Attribute Consistency](https://mlsanity.com/paper/2207.01011/) (2022)
- [BrainDecoder: Style-Based Visual Decoding of EEG Signals](https://mlsanity.com/paper/2409.05279/) (2024)
- [Schr"odinger Diffusion Driven Signal Recovery in 3T BOLD fMRI Using Unmatched 7T Observations](https://mlsanity.com/paper/2504.01004/) (2025)
- [MindShot: Multi-Shot Video Reconstruction from fMRI with LLM Decoding](https://mlsanity.com/paper/2508.02480/) (2025)
- [HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion](https://mlsanity.com/paper/2510.03122/) (2025)
- [SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction](https://mlsanity.com/paper/2601.17857/) (2026)
- [MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion](https://mlsanity.com/paper/2303.14139/) (2023)
- [Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning](https://mlsanity.com/paper/2404.05107/) (2024)
- [HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion](https://mlsanity.com/paper/2506.06035/) (2025)
- [NeuroSwift: A Lightweight Cross-Subject Framework for fMRI Visual Reconstruction of Complex Scenes](https://mlsanity.com/paper/2510.02266/) (2025)

## Abstract (arXiv)

Reconstructing seeing images from fMRI recordings is an absorbing research area in neuroscience and provides a potential brain-reading technology. The challenge lies in that visual encoding in brain is highly complex and not fully revealed. Inspired by the theory that visual features are hierarchically represented in cortex, we propose to break the complex visual signals into multi-level components and decode each component separately. Specifically, we decode shape and semantic representations from the lower and higher visual cortex respectively, and merge the shape and semantic information to images by a generative adversarial network (Shape-Semantic GAN). This 'divide and conquer' strategy captures visual information more accurately. Experiments demonstrate that Shape-Semantic GAN improves the reconstruction similarity and image quality, and achieves the state-of-the-art image reconstruction performance.