# Personalized visual encoding model construction with small data

arXiv: https://arxiv.org/abs/2202.02245 · HTML: https://mlsanity.com/paper/2202.02245/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 52 references that are in the map.
- First posted on arXiv 4 Feb 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks](https://mlsanity.com/paper/1810.03856/) (2018)
- [NeuroGen: activation optimized image synthesis for discovery neuroscience](https://mlsanity.com/paper/2105.07140/) (2021)
- [The extent and drivers of gender imbalance in neuroscience reference lists](https://mlsanity.com/paper/2001.01002/) (2020)
- [Fractional ridge regression: a fast, interpretable reparameterization of ridge regression](https://mlsanity.com/paper/2005.03220/) (2020)

## Built on by (most cited first, 12 of 12)

- [The Wisdom of a Crowd of Brains: A Universal Brain Encoder](https://mlsanity.com/paper/2406.12179/) (2024)
- [MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI](https://mlsanity.com/paper/2503.02351/) (2025)
- [A Framework for Interpretability in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2310.01685/) (2023)
- [SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning](https://mlsanity.com/paper/2508.10298/) (2025)
- [Generate your neural signals from mine: individual-to-individual EEG converters](https://mlsanity.com/paper/2304.10736/) (2023)
- [Across-subject ensemble-learning alleviates the need for large samples for fMRI decoding](https://mlsanity.com/paper/2407.12056/) (2024)
- [A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli](https://mlsanity.com/paper/2507.18104/) (2025)
- [How Hyper-Datafication Impacts the Sustainability Costs in Frontier AI](https://mlsanity.com/paper/2602.00056/) (2026)
- [Beyond Grid-Locked Voxels: Neural Response Functions for Continuous Brain Encoding](https://mlsanity.com/paper/2510.07342/) (2025)
- [NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity](https://mlsanity.com/paper/2604.09817/) (2026)
- [ViBE: Visual-to-M/EEG Brain Encoding via Spatio-Temporal VAE and Distribution-Aligned Projection](https://mlsanity.com/paper/2604.26218/) (2026)
- [BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language](https://mlsanity.com/paper/2606.30319/) (2026)

## Abstract (arXiv)

Encoding models that predict brain response patterns to stimuli are one way to capture this relationship between variability in bottom-up neural systems and individual's behavior or pathological state. However, they generally need a large amount of training data to achieve optimal accuracy. Here, we propose and test an alternative personalized ensemble encoding model approach to utilize existing encoding models, to create encoding models for novel individuals with relatively little stimuli-response data. We show that these personalized ensemble encoding models trained with small amounts of data for a specific individual, i.e. ~300 image-response pairs, achieve accuracy not different from models trained on ~20,000 image-response pairs for the same individual. Importantly, the personalized ensemble encoding models preserve patterns of inter-individual variability in the image-response relationship. Additionally, we show the proposed approach is robust against domain shift by validating on a prospectively collected set of image-response data in novel individuals with a different scanner and experimental setup. Finally, we use our personalized ensemble encoding model within the recently developed NeuroGen framework to generate optimal stimuli designed to maximize specific regions' activations for a specific individual. We show that the inter-individual differences in face areas responses to images of animal vs human faces observed previously is replicated using NeuroGen with the ensemble encoding model. Our approach shows the potential to use previously collected, deeply sampled data to efficiently create accurate, personalized encoding models and, subsequently, personalized optimal synthetic images for new individuals scanned under different experimental conditions.