# Neural Photofit: Gaze-based Mental Image Reconstruction

arXiv: https://arxiv.org/abs/2108.07524 · HTML: https://mlsanity.com/paper/2108.07524/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI](https://mlsanity.com/paper/1907.02431/) (2019)
- [Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN](https://mlsanity.com/paper/2001.11761/) (2020)
- [DCNN-GAN: Reconstructing Realistic Image from fMRI](https://mlsanity.com/paper/1901.07368/) (2019)
- [Prediction of Search Targets From Fixations in Open-World Settings](https://mlsanity.com/paper/1502.05137/) (2015)
- [Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning](https://mlsanity.com/paper/1906.12181/) (2019)
- [CG-GAN: An Interactive Evolutionary GAN-based Approach for Facial Composite Generation)](https://mlsanity.com/paper/1912.05020/) (2019)
- [Multi-task Generative Adversarial Learning on Geometrical Shape Reconstruction from EEG Brain Signals](https://mlsanity.com/paper/1907.13351/) (2019)

## Built on by (most cited first, 5 of 5)

- [Learning User Embeddings from Human Gaze for Personalised Saliency Prediction](https://mlsanity.com/paper/2403.13653/) (2024)
- [Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenes](https://mlsanity.com/paper/2508.01853/) (2025)
- [Implicit Search Intent Recognition using EEG and Eye Tracking: Novel Dataset and Cross-User Prediction](https://mlsanity.com/paper/2508.01860/) (2025)
- [HAIFAI: Human-AI Interaction for Mental Face Reconstruction](https://mlsanity.com/paper/2412.06323/) (2024)
- [Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention](https://mlsanity.com/paper/2607.23917/) (2026)

## Abstract (arXiv)

We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and neural attention and predicts a relevance score for each extracted image feature. Finally, image features are aggregated into a single feature vector as a linear combination of all features weighted by relevance which a decoder decodes into the final photofit. We train the neural scoring network on a novel dataset containing gaze data of 19 participants looking at collages of synthetic faces. We show that our method significantly outperforms a mean baseline predictor and report on a human study that shows that we can decode photofits that are visually plausible and close to the observer's mental image.