# A foundation model of vision, audition, and language for in-silico neuroscience

arXiv: https://arxiv.org/abs/2605.04326 · HTML: https://mlsanity.com/paper/2605.04326/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 108 references that are in the map.
- First posted on arXiv 5 May 2026.

## Builds on (most cited first, 30 of 35)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Learnable latent embeddings for joint behavioral and neural analysis](https://mlsanity.com/paper/2204.00673/) (2022)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [Toward a realistic model of speech processing in the brain with self-supervised learning](https://mlsanity.com/paper/2206.01685/) (2022)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [Brain encoding models based on multimodal transformers can transfer across language and vision](https://mlsanity.com/paper/2305.12248/) (2023)
- [UMBRAE: Unified Multimodal Brain Decoding](https://mlsanity.com/paper/2404.07202/) (2024)
- [Scaling laws for decoding images from brain activity](https://mlsanity.com/paper/2501.15322/) (2025)
- [The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies](https://mlsanity.com/paper/2501.00504/) (2024)
- [TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction](https://mlsanity.com/paper/2507.22229/) (2025)
- [The Wisdom of a Crowd of Brains: A Universal Brain Encoder](https://mlsanity.com/paper/2406.12179/) (2024)
- [OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning](https://mlsanity.com/paper/2507.13364/) (2025)
- [Visio-Linguistic Brain Encoding](https://mlsanity.com/paper/2204.08261/) (2022)
- [Modeling the dynamics of human brain activity with recurrent neural networks](https://mlsanity.com/paper/1606.03071/) (2016)
- [The Algonauts Project 2023 Challenge: UARK-UAlbany Team Solution](https://mlsanity.com/paper/2308.00262/) (2023)
- [Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain](https://mlsanity.com/paper/2311.07766/) (2023)
- [VIBE: Video-Input Brain Encoder for fMRI Response Modeling](https://mlsanity.com/paper/2507.17958/) (2025)
- [Extracting representations of cognition across neuroimaging studies improves brain decoding](https://mlsanity.com/paper/1809.06035/) (2018)
- [Memory Encoding Model](https://mlsanity.com/paper/2308.01175/) (2023)

## Built on by (most cited first, 13 of 13)

- [NeuralBench: A Unifying Framework to Benchmark NeuroAI Models](https://mlsanity.com/paper/2605.08495/) (2026)
- [Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings](https://mlsanity.com/paper/2608.18114/) (2026)
- [Attention Alignment Between Humans and Vision-Language Models](https://mlsanity.com/paper/2606.17410/) (2026)
- [Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation](https://mlsanity.com/paper/2606.06345/) (2026)
- [Flow Matching with In-Context Priors for Out-of-Distribution Brain Dynamics](https://mlsanity.com/paper/2606.11833/) (2026)
- [Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2](https://mlsanity.com/paper/2605.13904/) (2026)
- [Platonic Projection Structures: Operator-Induced Observability in Representation Learning](https://mlsanity.com/paper/2607.05175/) (2026)
- [Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis](https://mlsanity.com/paper/2608.19902/) (2026)
- [The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models](https://mlsanity.com/paper/2609.10947/) (2026)
- [Can a Neural Encoding Model Replicate an fMRI Visualization Study?](https://mlsanity.com/paper/2609.15685/) (2026)
- [Cognitive Expert Language Models Better Align with the Corresponding Brain Systems](https://mlsanity.com/paper/2609.36239/) (2026)
- [EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights](https://mlsanity.com/paper/2610.00492/) (2026)
- [What We Talk About When We Talk About Emotion Editing](https://mlsanity.com/paper/2610.13558/) (2026)

## Abstract (arXiv)

Cognitive neuroscience is fragmented into specialized models, each tailored to specific experimental paradigms, hence preventing a unified model of cognition in the human brain. Here, we introduce TRIBE v2, a tri-modal (video, audio and language) foundation model capable of predicting human brain activity in a variety of naturalistic and experimental conditions. Leveraging a unified dataset of over 1,000 hours of fMRI across 720 subjects, we demonstrate that our model accurately predicts high-resolution brain responses for novel stimuli, tasks and subjects, superseding traditional linear encoding models, delivering several-fold improvements in accuracy. Critically, TRIBE v2 enables in silico experimentation: tested on seminal visual and neuro-linguistic paradigms, it recovers a variety of results established by decades of empirical research. Finally, by extracting interpretable latent features, TRIBE v2 reveals the fine-grained topography of multisensory integration. These results establish artificial intelligence as a unifying framework for exploring the functional organization of the human brain.