# Improving Multimodal Brain Encoding Model with Dynamic Subject-awareness Routing

arXiv: https://arxiv.org/abs/2510.04670 · HTML: https://mlsanity.com/paper/2510.04670/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Brain encoding models based on multimodal transformers can transfer across language and vision](https://mlsanity.com/paper/2305.12248/) (2023)
- [UMBRAE: Unified Multimodal Brain Decoding](https://mlsanity.com/paper/2404.07202/) (2024)
- [The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies](https://mlsanity.com/paper/2501.00504/) (2024)
- [TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction](https://mlsanity.com/paper/2507.22229/) (2025)
- [MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data](https://mlsanity.com/paper/2502.05034/) (2025)
- [MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI](https://mlsanity.com/paper/2503.02351/) (2025)
- [MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding](https://mlsanity.com/paper/2502.15786/) (2025)
- [Multi-modal brain encoding models for multi-modal stimuli](https://mlsanity.com/paper/2505.20027/) (2025)
- [BP-GPT: Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2502.15172/) (2025)
- [A Bi-Pyramid Multimodal Fusion Method for the Diagnosis of Bipolar Disorders](https://mlsanity.com/paper/2401.07571/) (2024)

## Built on by (most cited first, 2 of 2)

- [NeuroSeg Meets DINOv3: Transferring 2D Self-Supervised Visual Priors to 3D Neuron Segmentation via DINOv3 Initialization](https://mlsanity.com/paper/2603.23104/) (2026)
- [BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction](https://mlsanity.com/paper/2602.04512/) (2026)

## Abstract (arXiv)

Naturalistic fMRI encoding must handle multimodal inputs, shifting fusion styles, and pronounced inter-subject variability. We introduce AFIRE (Agnostic Framework for Multimodal fMRI Response Encoding), an agnostic interface that standardizes time-aligned post-fusion tokens from varied encoders, and MIND, a plug-and-play Mixture-of-Experts decoder with a subject-aware dynamic gating. Trained end-to-end for whole-brain prediction, AFIRE decouples the decoder from upstream fusion, while MIND combines token-dependent Top-K sparse routing with a subject prior to personalize expert usage without sacrificing generality. Experiments across multiple multimodal backbones and subjects show consistent improvements over strong baselines, enhanced cross-subject generalization, and interpretable expert patterns that correlate with content type. The framework offers a simple attachment point for new encoders and datasets, enabling robust, plug-and-improve performance for naturalistic neuroimaging studies.