mlsanityData to 7 Oct 2026

A foundation model of vision, audition, and language for in-silico neuroscience

arXiv 2605.04326 ↗First posted 5 May 2026q-bio.NC cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Cognitive neuroscience is fragmented into specialized models, each tailored to specific experimental paradigms, hence preventing a unified model of cognition in the human brain. Here, we introduce TRIBE v2, a tri-modal (video, audio and language) foundation model capable of predicting human brain activity in a variety of naturalistic and experimental conditions. Leveraging a unified dataset of over 1,000 hours of fMRI across 720 subjects, we demonstrate that our model accurately predicts high-resolution brain responses for novel stimuli, tasks and subjects, superseding traditional linear encoding models, delivering several-fold improvements in accuracy. Critically, TRIBE v2 enables in silico experimentation: tested on seminal visual and neuro-linguistic paradigms, it recovers a variety of results established by decades of empirical research. Finally, by extracting interpretable latent features, TRIBE v2 reveals the fine-grained topography of multisensory integration. These results establish artificial intelligence as a unifying framework for exploring the functional organization of the human brain.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 108 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

35 of 108 references.

References · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  4. 2024Phi-4 Technical Reportcited by 675
  5. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  6. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  7. 2022Decoding speech perception from non-invasive brain recordingscited by 129
  8. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  9. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
  10. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  11. 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
  12. 2023Scaling laws for language encoding models in fMRIcited by 68
23 more
  1. 2023Brain decoding: toward real-time reconstruction of visual perceptioncited by 64
  2. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  3. 2023A Unified, Scalable Framework for Neural Population Decodingcited by 59
  4. 2016Neural Encoding and Decoding with Deep Learning for Dynamic Natural Visioncited by 58
  5. 2023Brain encoding models based on multimodal transformers can transfer across language and visioncited by 31
  6. 2024UMBRAE: Unified Multimodal Brain Decodingcited by 31
  7. 2025Scaling laws for decoding images from brain activitycited by 21
  8. 2024The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Moviescited by 19
  9. 2025TRIBE: TRImodal Brain Encoder for whole-brain fMRI response predictioncited by 17
  10. 2024The Wisdom of a Crowd of Brains: A Universal Brain Encodercited by 15
  11. 2022Visio-Linguistic Brain Encodingcited by 13
  12. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 13
  13. 2016Modeling the dynamics of human brain activity with recurrent neural networkscited by 12
  14. 2023The Algonauts Project 2023 Challenge: UARK-UAlbany Team Solutioncited by 11
  15. 2023Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Braincited by 9
  16. 2018Extracting representations of cognition across neuroimaging studies improves brain decodingcited by 7
  17. 2023Memory Encoding Modelcited by 7
  18. 2025VIBE: Video-Input Brain Encoder for fMRI Response Modelingcited by 7
  19. 2025BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Languagecited by 6
  20. 2025Multimodal Recurrent Ensembles for Predicting Brain Responses to Naturalistic Movies (Algonauts 2025)cited by 5
  21. 2025Predicting Brain Responses To Natural Movies With Multimodal LLMscited by 5
  22. 2024In silico discovery of representational relationships across visual cortexcited by 3
  23. 2025SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experimentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026NeuralBench: A Unifying Framework to Benchmark NeuroAI Modelscited by 13
  2. 2026Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordingscited by 3
  3. 2026Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentationcited by 1
  4. 2026Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2cited by 0
  5. 2026Flow Matching with In-Context Priors for Out-of-Distribution Brain Dynamicscited by 0
  6. 2026Attention Alignment Between Humans and Vision-Language Modelscited by 0
  7. 2026Platonic Projection Structures: Operator-Induced Observability in Representation Learningcited by 0
  8. 2026Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysiscited by 0
  9. 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language modelscited by 0
  10. 2026Can a Neural Encoding Model Replicate an fMRI Visualization Study?cited by 0
  11. 2026Cognitive Expert Language Models Better Align with the Corresponding Brain Systemscited by 0
  12. 2026EurekaBench: Measuring Agentic Ability to Discover New Scientific Insightscited by 0