# MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

arXiv: https://arxiv.org/abs/2605.29850 · HTML: https://mlsanity.com/paper/2605.29850/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes](https://mlsanity.com/paper/2301.03198/) (2023)
- [Brain encoding models based on multimodal transformers can transfer across language and vision](https://mlsanity.com/paper/2305.12248/) (2023)
- [From Language to Cognition: How LLMs Outgrow the Human Language Network](https://mlsanity.com/paper/2503.01830/) (2025)
- [Improving Semantic Understanding in Speech Language Models via Brain-tuning](https://mlsanity.com/paper/2410.09230/) (2024)
- [The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies](https://mlsanity.com/paper/2501.00504/) (2024)
- [TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction](https://mlsanity.com/paper/2507.22229/) (2025)
- [The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion](https://mlsanity.com/paper/2104.13714/) (2021)
- [Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream](https://mlsanity.com/paper/2411.05712/) (2024)
- [The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence](https://mlsanity.com/paper/1905.05675/) (2019)
- [Beyond linear regression: mapping models in cognitive neuroscience should align with research goals](https://mlsanity.com/paper/2208.10668/) (2022)
- [Transformer brain encoders explain human high-level visual responses](https://mlsanity.com/paper/2505.17329/) (2025)
- [Multi-modal brain encoding models for multi-modal stimuli](https://mlsanity.com/paper/2505.20027/) (2025)
- [VIBE: Video-Input Brain Encoder for fMRI Response Modeling](https://mlsanity.com/paper/2507.17958/) (2025)
- [Predicting Brain Responses To Natural Movies With Multimodal LLMs](https://mlsanity.com/paper/2507.19956/) (2025)
- [Multimodal Recurrent Ensembles for Predicting Brain Responses to Naturalistic Movies (Algonauts 2025)](https://mlsanity.com/paper/2507.17897/) (2025)
- [In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain](https://mlsanity.com/paper/2510.21142/) (2025)
- [Modeling the Human Visual System: Comparative Insights from Response-Optimized and Task-Optimized Vision Models, Language Models, and different Readout Mechanisms](https://mlsanity.com/paper/2410.14031/) (2024)
- [A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli](https://mlsanity.com/paper/2507.18104/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Platonic brain bridge hypothesis: human brain networks as an architectural prior for multimodal large language models](https://mlsanity.com/paper/2609.10947/) (2026)

## Abstract (arXiv)

Recent progress in task-optimized neural networks has established encoding models as a powerful tool for predicting brain responses to naturalistic stimuli, yet most existing approaches rely on unimodal representations. The emergence of omni-modal foundation models and rich multimodal neural datasets enables encoding models that jointly integrate visual, auditory, and linguistic information across subjects. We introduce MIRAGE, a brain encoding framework for predicting whole-brain fMRI responses to naturalistic audiovisual stimuli. MIRAGE achieves state-of-the-art performance via a native multimodal backbone and adaptive feature gating across layers. These representations are then combined with a transformer-based brain encoder and a subject-specific linear head over the cortical parcels. Controlled comparisons show that natively multimodal features consistently outperform post-hoc aggregation of independent unimodal features, across architectural levels and backbones. Beyond predictive accuracy, the learned attention weights are directly inspectable to interpret the modality-specific gating profile over the backbone, and each modality traces a distinct anatomical pattern across cortex. Together, these results propose adaptive layer-wise aggregation of natively multimodal features as a generalizable, interpretable, and accurate approach for whole-brain encoding.