# NeuroGen: activation optimized image synthesis for discovery neuroscience

arXiv: https://arxiv.org/abs/2105.07140 · HTML: https://mlsanity.com/paper/2105.07140/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 64 references that are in the map.
- First posted on arXiv 15 May 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN](https://mlsanity.com/paper/2001.11761/) (2020)
- [Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks](https://mlsanity.com/paper/1810.03856/) (2018)
- [Robust Optimization for Deep Regression](https://mlsanity.com/paper/1505.06606/) (2015)
- [The extent and drivers of gender imbalance in neuroscience reference lists](https://mlsanity.com/paper/2001.01002/) (2020)
- [Fractional ridge regression: a fast, interpretable reparameterization of ridge regression](https://mlsanity.com/paper/2005.03220/) (2020)

## Built on by (most cited first, 22 of 22)

- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs](https://mlsanity.com/paper/2202.12692/) (2022)
- [Decoding natural image stimuli from fMRI data with a surface-based convolutional network](https://mlsanity.com/paper/2212.02409/) (2022)
- [NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation](https://mlsanity.com/paper/2403.18211/) (2024)
- [Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models](https://mlsanity.com/paper/2306.03089/) (2023)
- [BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity](https://mlsanity.com/paper/2310.04420/) (2023)
- [Brain Decodes Deep Nets](https://mlsanity.com/paper/2312.01280/) (2023)
- [Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers](https://mlsanity.com/paper/2410.05266/) (2024)
- [Personalized visual encoding model construction with small data](https://mlsanity.com/paper/2202.02245/) (2022)
- [Applications of Generative Adversarial Networks in Neuroimaging and Clinical Neuroscience](https://mlsanity.com/paper/2206.07081/) (2022)
- [ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding](https://mlsanity.com/paper/2510.27128/) (2025)
- [Reanimating Images using Neural Representations of Dynamic Stimuli](https://mlsanity.com/paper/2406.02659/) (2024)
- [BrainExplore: Large-Scale Discovery of Interpretable Visual Representations in the Human Brain](https://mlsanity.com/paper/2512.08560/) (2025)
- [Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex](https://mlsanity.com/paper/2505.15813/) (2025)
- [Retinotopy Inspired Brain Encoding Model and the All-for-One Training Recipe](https://mlsanity.com/paper/2307.14021/) (2023)
- [Meta-learning In-Context Enables Training-Free Cross Subject Brain Decoding](https://mlsanity.com/paper/2604.08537/) (2026)
- [NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity](https://mlsanity.com/paper/2607.02317/) (2026)
- [Visualizing and Controlling Cortical Responses Using Voxel-Weighted Activation Maximization](https://mlsanity.com/paper/2506.04379/) (2025)
- [NeuroVolve: Evolving Visual Stimuli toward Programmable Neural Objectives](https://mlsanity.com/paper/2512.00557/) (2025)
- [The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling](https://mlsanity.com/paper/2601.15071/) (2026)
- [From Activation to Specificity: Automating Counterfactual Testing of Visual Representations in the Human Brain](https://mlsanity.com/paper/2605.23895/) (2026)

## Abstract (arXiv)

Functional MRI (fMRI) is a powerful technique that has allowed us to characterize visual cortex responses to stimuli, yet such experiments are by nature constructed based on a priori hypotheses, limited to the set of images presented to the individual while they are in the scanner, are subject to noise in the observed brain responses, and may vary widely across individuals. In this work, we propose a novel computational strategy, which we call NeuroGen, to overcome these limitations and develop a powerful tool for human vision neuroscience discovery. NeuroGen combines an fMRI-trained neural encoding model of human vision with a deep generative network to synthesize images predicted to achieve a target pattern of macro-scale brain activation. We demonstrate that the reduction of noise that the encoding model provides, coupled with the generative network's ability to produce images of high fidelity, results in a robust discovery architecture for visual neuroscience. By using only a small number of synthetic images created by NeuroGen, we demonstrate that we can detect and amplify differences in regional and individual human brain response patterns to visual stimuli. We then verify that these discoveries are reflected in the several thousand observed image responses measured with fMRI. We further demonstrate that NeuroGen can create synthetic images predicted to achieve regional response patterns not achievable by the best-matching natural images. The NeuroGen framework extends the utility of brain encoding models and opens up a new avenue for exploring, and possibly precisely controlling, the human visual system.