# Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE

arXiv: https://arxiv.org/abs/2606.15989 · HTML: https://mlsanity.com/paper/2606.15989/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes](https://mlsanity.com/paper/2301.03198/) (2023)
- [Brain encoding models based on multimodal transformers can transfer across language and vision](https://mlsanity.com/paper/2305.12248/) (2023)
- [Through their eyes: multi-subject Brain Decoding with simple alignment techniques](https://mlsanity.com/paper/2309.00627/) (2023)
- [MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data](https://mlsanity.com/paper/2502.05034/) (2025)
- [UniBrain: A Unified Model for Cross-Subject Brain Decoding](https://mlsanity.com/paper/2412.19487/) (2024)
- [Inter-individual and inter-site neural code conversion without shared stimuli](https://mlsanity.com/paper/2403.11517/) (2024)

## Abstract (arXiv)

Aligning neural activity across subjects offers the promise of discovering shared computational principles and generalizable decoders. However, traditional alignment methods require shared stimuli across subjects, a constraint that limits applicability to naturalistic paradigms with limited or non-overlapping data. We introduce a Multi-Encoder-Decoder Variational Autoencoder (MED-VAE) that achieves cross-subject alignment without shared stimuli by anchoring representations to a common scaffold provided by a pretrained ANN. Using the Natural Scenes Dataset, we show that MED-VAE creates common latent spaces with superior semantic organisation, achieving higher cross-subject alignment than common methods while maintaining robust generalisation to held-out stimuli where traditional methods degrade. Reconstructing from these common spaces back to each subject's original neural space, MED-VAE preserves equal stimulus-driven signal in its cross-subject latent space. Finally, we show that this superior alignment directly enables cross-subject neural prediction, as demonstrated via cross-subject image decoding. In summary, we introduce a framework to identify generalisable common subspaces for cross-subject predictions and downstream tasks, demonstrated here for visual cortex responses to static images.