# Through their eyes: multi-subject Brain Decoding with simple alignment techniques

arXiv: https://arxiv.org/abs/2309.00627 · HTML: https://mlsanity.com/paper/2309.00627/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 56 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 19 of 19)

- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Instance-Conditioned GAN](https://mlsanity.com/paper/2109.05070/) (2021)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [BOLD5000: A public fMRI dataset of 5000 images](https://mlsanity.com/paper/1809.01281/) (2018)
- [Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks](https://mlsanity.com/paper/1810.03856/) (2018)
- [BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding](https://mlsanity.com/paper/2302.12971/) (2023)
- [Brain Captioning: Decoding human brain activity into images and text](https://mlsanity.com/paper/2305.11560/) (2023)
- [Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models](https://mlsanity.com/paper/2306.03089/) (2023)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli](https://mlsanity.com/paper/2212.06726/) (2022)
- [Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning](https://mlsanity.com/paper/1906.12181/) (2019)
- [Fast shared response model for fMRI data](https://mlsanity.com/paper/1909.12537/) (2019)

## Built on by (most cited first, 21 of 21)

- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction](https://mlsanity.com/paper/2404.12630/) (2024)
- [NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction](https://mlsanity.com/paper/2410.19452/) (2024)
- [NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery](https://mlsanity.com/paper/2506.06898/) (2025)
- [A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli](https://mlsanity.com/paper/2503.15978/) (2025)
- [MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding](https://mlsanity.com/paper/2505.15946/) (2025)
- [ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding](https://mlsanity.com/paper/2510.27128/) (2025)
- [A Survey of fMRI to Image Reconstruction](https://mlsanity.com/paper/2502.16861/) (2025)
- [BrainBits: How Much of the Brain are Generative Reconstruction Methods Using?](https://mlsanity.com/paper/2411.02783/) (2024)
- [Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer](https://mlsanity.com/paper/2510.25976/) (2025)
- [R&B -- Rhythm and Brain: Cross-subject Decoding of Music from Human Brain Activity](https://mlsanity.com/paper/2406.15537/) (2024)
- [Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representations](https://mlsanity.com/paper/2505.01670/) (2025)
- [Towards Interpretable Visual Decoding with Attention to Brain Representations](https://mlsanity.com/paper/2509.23566/) (2025)
- [Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding](https://mlsanity.com/paper/2603.07625/) (2026)
- [Simple Models, Rich Representations: Visual Decoding from Primate Intracortical Neural Signals](https://mlsanity.com/paper/2601.11108/) (2026)
- [Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data](https://mlsanity.com/paper/2604.15374/) (2026)
- [StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation](https://mlsanity.com/paper/2605.02586/) (2026)
- [MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery](https://mlsanity.com/paper/2605.17198/) (2026)
- [Brain-IT-VQA: From Brain Signals to Answers](https://mlsanity.com/paper/2605.29588/) (2026)
- [Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE](https://mlsanity.com/paper/2606.15989/) (2026)
- [Real-time Reconstruction of Human Visual Perception from fMRI](https://mlsanity.com/paper/2607.22753/) (2026)

## Abstract (arXiv)

Previous brain decoding research primarily involves single-subject studies, reconstructing stimuli via fMRI activity from the same subject. Our study aims to introduce a generalization technique for cross-subject brain decoding, facilitated by exploring data alignment methods. We utilized the NSD dataset, a comprehensive 7T fMRI vision experiment involving multiple subjects exposed to 9841 images, 982 of which were viewed by all. Our approach involved training a decoding model on one subject, aligning others' data to this space, and testing the decoding on the second subject. We compared ridge regression, hyper alignment, and anatomical alignment techniques for fMRI data alignment. We established that cross-subject brain decoding is feasible, even using around 10% of the total data, or 982 common images, with comparable performance to single-subject decoding. Ridge regression was the best method for functional alignment. Through subject alignment, we achieved superior brain decoding and a potential 90% reduction in scan time. This could pave the way for more efficient experiments and further advancements in the field, typically requiring an exorbitant 20-hour scan time per subject.