mlsanityData to 14 Oct 2026

VIBE: Video-Input Brain Encoder for fMRI Response Modeling

arXiv 2507.17958 ↗First posted 23 Jul 2025cs.LG cs.AI cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

We present VIBE, a two-stage Transformer that fuses multi-modal video, audio, and text features to predict fMRI activity. Representations from open-source models (Qwen2.5, BEATs, Whisper, SlowFast, V-JEPA) are merged by a modality-fusion transformer and temporally decoded by a prediction transformer with rotary embeddings. Trained on 65 hours of movie data from the CNeuroMod dataset and ensembled across 20 seeds, VIBE attains mean parcel-wise Pearson correlations of 0.3225 on in-distribution Friends S07 and 0.2125 on six out-of-distribution films. An earlier iteration of the same architecture obtained 0.3198 and 0.2096, respectively, winning Phase-1 and placing second overall in the Algonauts 2025 Challenge.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 8 of its 22 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

8 of 22 references.

References · 8

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2018SlowFast Networks for Video Recognitioncited by 1,867
  4. 2025Qwen2.5-Omni Technical Reportcited by 736
  5. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  6. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  7. 2024The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Moviescited by 20
  8. 2025Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Unitscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2026A foundation model of vision, audition, and language for in-silico neurosciencecited by 13
  2. 2025Insights from the Algonauts 2025 Winnerscited by 4
  3. 2026Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Modelcited by 2
  4. 2025Stacked Regression using Off-the-shelf, Stimulus-tuned and Fine-tuned Neural Networks for Predicting fMRI Brain Responses to Movies (Algonauts 2025 Report)cited by 1
  5. 2026MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encodingcited by 1
  6. 2025Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perceptioncited by 0
  7. 2026Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matchingcited by 0
  8. 2026MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matchingcited by 0