# Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs

arXiv: https://arxiv.org/abs/2306.11536 · HTML: https://mlsanity.com/paper/2306.11536/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [Decoding natural image stimuli from fMRI data with a surface-based convolutional network](https://mlsanity.com/paper/2212.02409/) (2022)
- [BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding](https://mlsanity.com/paper/2302.12971/) (2023)
- [Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities](https://mlsanity.com/paper/2305.17214/) (2023)
- [Brain Captioning: Decoding human brain activity into images and text](https://mlsanity.com/paper/2305.11560/) (2023)
- [Controllable Mind Visual Diffusion Model](https://mlsanity.com/paper/2305.10135/) (2023)

## Built on by (most cited first, 23 of 23)

- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [DREAM: Visual Decoding from Reversing Human Visual System](https://mlsanity.com/paper/2310.02265/) (2023)
- [UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity](https://mlsanity.com/paper/2308.07428/) (2023)
- [UMBRAE: Unified Multimodal Brain Decoding](https://mlsanity.com/paper/2404.07202/) (2024)
- [BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction](https://mlsanity.com/paper/2312.14871/) (2023)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [Brain2Music: Reconstructing Music from Human Brain Activity](https://mlsanity.com/paper/2307.11078/) (2023)
- [Brain-optimized inference improves reconstructions of fMRI brain activity](https://mlsanity.com/paper/2312.07705/) (2023)
- [NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery](https://mlsanity.com/paper/2506.06898/) (2025)
- [MindGPT: Interpreting What You See with Non-invasive Brain Recordings](https://mlsanity.com/paper/2309.15729/) (2023)
- [BrainBits: How Much of the Brain are Generative Reconstruction Methods Using?](https://mlsanity.com/paper/2411.02783/) (2024)
- [How does the primate brain combine generative and discriminative computations in vision?](https://mlsanity.com/paper/2401.06005/) (2024)
- [MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding](https://mlsanity.com/paper/2405.17720/) (2024)
- [Exploring The Visual Feature Space for Multimodal Neural Decoding](https://mlsanity.com/paper/2505.15755/) (2025)
- [Learning Brain Representation with Hierarchical Visual Embeddings](https://mlsanity.com/paper/2602.07495/) (2026)
- [Autoregressive Visual Decoding from EEG Signals](https://mlsanity.com/paper/2602.22555/) (2026)
- [MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model](https://mlsanity.com/paper/2405.18812/) (2024)
- [Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing](https://mlsanity.com/paper/2503.10663/) (2025)
- [Multigranular Evaluation for Brain Visual Decoding](https://mlsanity.com/paper/2507.07993/) (2025)
- [ENIGMA: EEG-to-Image in 15 Minutes Using Less Than 1% of the Parameters](https://mlsanity.com/paper/2602.10361/) (2026)
- [Unified Multimodal Brain Decoding via Cross-Subject Soft-ROI Fusion](https://mlsanity.com/paper/2512.20249/) (2025)
- [MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery](https://mlsanity.com/paper/2605.17198/) (2026)

## Abstract (arXiv)

The integration of deep learning and neuroscience has been advancing rapidly, which has led to improvements in the analysis of brain activity and the understanding of deep learning models from a neuroscientific perspective. The reconstruction of visual experience from human brain activity is an area that has particularly benefited: the use of deep learning models trained on large amounts of natural images has greatly improved its quality, and approaches that combine the diverse information contained in visual experiences have proliferated rapidly in recent years. In this technical paper, by taking advantage of the simple and generic framework that we proposed (Takagi and Nishimoto, CVPR 2023), we examine the extent to which various additional decoding techniques affect the performance of visual experience reconstruction. Specifically, we combined our earlier work with the following three techniques: using decoded text from brain activity, nonlinear optimization for structural image reconstruction, and using decoded depth information from brain activity. We confirmed that these techniques contributed to improving accuracy over the baseline. We also discuss what researchers should consider when performing visual reconstruction using deep generative models trained on large datasets. Please check our webpage at https://sites.google.com/view/stablediffusion-with-brain/. Code is also available at https://github.com/yu-takagi/StableDiffusionReconstruction.