# VIBE: Video-Input Brain Encoder for fMRI Response Modeling

arXiv: https://arxiv.org/abs/2507.17958 · HTML: https://mlsanity.com/paper/2507.17958/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Builds on (most cited first, 8 of 8)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies](https://mlsanity.com/paper/2501.00504/) (2024)
- [Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Units](https://mlsanity.com/paper/2506.19732/) (2025)

## Built on by (most cited first, 8 of 8)

- [A foundation model of vision, audition, and language for in-silico neuroscience](https://mlsanity.com/paper/2605.04326/) (2026)
- [Insights from the Algonauts 2025 Winners](https://mlsanity.com/paper/2508.10784/) (2025)
- [Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Model](https://mlsanity.com/paper/2606.09770/) (2026)
- [Stacked Regression using Off-the-shelf, Stimulus-tuned and Fine-tuned Neural Networks for Predicting fMRI Brain Responses to Movies (Algonauts 2025 Report)](https://mlsanity.com/paper/2510.06235/) (2025)
- [MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding](https://mlsanity.com/paper/2605.29850/) (2026)
- [Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception](https://mlsanity.com/paper/2510.01502/) (2025)
- [Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matching](https://mlsanity.com/paper/2604.11178/) (2026)
- [MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching](https://mlsanity.com/paper/2610.15339/) (2026)

## Abstract (arXiv)

We present VIBE, a two-stage Transformer that fuses multi-modal video, audio, and text features to predict fMRI activity. Representations from open-source models (Qwen2.5, BEATs, Whisper, SlowFast, V-JEPA) are merged by a modality-fusion transformer and temporally decoded by a prediction transformer with rotary embeddings. Trained on 65 hours of movie data from the CNeuroMod dataset and ensembled across 20 seeds, VIBE attains mean parcel-wise Pearson correlations of 0.3225 on in-distribution Friends S07 and 0.2125 on six out-of-distribution films. An earlier iteration of the same architecture obtained 0.3198 and 0.2096, respectively, winning Phase-1 and placing second overall in the Algonauts 2025 Challenge.