mlsanityData to 14 Oct 2026

TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction

arXiv 2507.22229 ↗First posted 29 Jul 2025cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Historically, neuroscience has progressed by fragmenting into specialized domains, each focusing on isolated modalities, tasks, or brain regions. While fruitful, this approach hinders the development of a unified model of cognition. Here, we introduce TRIBE, the first deep neural network trained to predict brain responses to stimuli across multiple modalities, cortical areas and individuals. By combining the pretrained representations of text, audio and video foundational models and handling their time-evolving nature with a transformer, our model can precisely model the spatial and temporal fMRI responses to videos, achieving the first place in the Algonauts 2025 brain encoding competition with a significant margin over competitors. Ablations show that while unimodal models can reliably predict their corresponding cortical networks (e.g. visual or auditory networks), they are systematically outperformed by our multimodal model in high-level associative cortices. Currently applied to perception and comprehension, our approach paves the way towards building an integrative model of representations in the human brain. Our code is available at https://github.com/facebookresearch/algonauts-2025.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 21 of its 48 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

21 of 48 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  7. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  8. 2024Phi-4 Technical Reportcited by 711
  9. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  11. 2022Decoding speech perception from non-invasive brain recordingscited by 130
  12. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
9 more
  1. 2023Scaling laws for language encoding models in fMRIcited by 68
  2. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  3. 2023Brain encoding models based on multimodal transformers can transfer across language and visioncited by 32
  4. 2024The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Moviescited by 20
  5. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 14
  6. 2022Visio-Linguistic Brain Encodingcited by 13
  7. 2016Modeling the dynamics of human brain activity with recurrent neural networkscited by 12
  8. 2023Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Braincited by 9
  9. 2025SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experimentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2026A foundation model of vision, audition, and language for in-silico neurosciencecited by 13
  2. 2025Insights from the Algonauts 2025 Winnerscited by 4
  3. 2026OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokenscited by 4
  4. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  5. 2025Improving Multimodal Brain Encoding Model with Dynamic Subject-awareness Routingcited by 2
  6. 2026Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Modelcited by 2
  7. 2025Stacked Regression using Off-the-shelf, Stimulus-tuned and Fine-tuned Neural Networks for Predicting fMRI Brain Responses to Movies (Algonauts 2025 Report)cited by 1
  8. 2025The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friendscited by 1
  9. 2025From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretrainingcited by 1
  10. 2026MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encodingcited by 1
  11. 2026NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivitycited by 1
  12. 2026BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Predictioncited by 0
7 more
  1. 2026Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matchingcited by 0
  2. 2026Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2cited by 0
  3. 2026EmoMind: Decoding Affective Captions from Human Brain fMRIcited by 0
  4. 2026MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decodingcited by 0
  5. 2026A global predicted-fMRI drive signal from TRIBE does not predict YouTube replay heatmapscited by 0
  6. 2026The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language modelscited by 0
  7. 2026MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matchingcited by 0