# BrainTaskonomy: Learning How to Pretrain and What to Transfer in fMRI Foundation Models

arXiv: https://arxiv.org/abs/2609.10518 · HTML: https://mlsanity.com/paper/2609.10518/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [RegMix: Data Mixture as Regression for Language Model Pre-training](https://mlsanity.com/paper/2407.01492/) (2024)
- [Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models](https://mlsanity.com/paper/2307.14430/) (2023)
- [DoGE: Domain Reweighting with Generalization Estimation](https://mlsanity.com/paper/2310.15393/) (2023)
- [Denoising Task Difficulty-based Curriculum for Training Diffusion Models](https://mlsanity.com/paper/2403.10348/) (2024)
- [CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing](https://mlsanity.com/paper/2503.06940/) (2025)
- [Omni-fMRI: A Universal Atlas-Free fMRI Foundation Model](https://mlsanity.com/paper/2601.23090/) (2026)
- [Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining](https://mlsanity.com/paper/2604.12683/) (2026)
- [Uncovering cognitive taskonomy through transfer learning in masked autoencoder-based fMRI reconstruction](https://mlsanity.com/paper/2407.00033/) (2024)
- [FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI](https://mlsanity.com/paper/2606.11500/) (2026)
- [Adopting a human developmental visual diet yields robust, shape-based AI vision](https://mlsanity.com/paper/2507.03168/) (2025)
- [BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics](https://mlsanity.com/paper/2606.17742/) (2026)
- [Beyond Single-Source Cognitive Taskonomy:Multi-Source Task Relations through fMRI Transfer Learning](https://mlsanity.com/paper/2606.26279/) (2026)

## Built on by (most cited first, 1 of 1)

- [A Scaling Study for fMRI Foundation Models](https://mlsanity.com/paper/2609.27232/) (2026)

## Abstract (arXiv)

fMRI foundation models increasingly aggregate heterogeneous data across brain states, cohorts, and acquisition settings, yet pretraining domains are commonly treated as a flat mixture and downstream tasks are adapted independently. We study whether measured learning relations can organize both stages without modifying the backbone. During pretraining, a lightweight Brain-DiT proxy estimates difficulty and directed facilitation across ten fMRI domains, yielding a priority-guided cumulative domain curriculum combined with high-to-low-noise timestep scheduling and joint consolidation. During adaptation, controlled first- and higher-order transfer across fifteen tasks constructs a directed taskonomy, from which budgeted integer programming (BIP) selects directly supervised source tasks and target-specific routes. The joint priority-domain and high-to-low-timestep curriculum reduces v-NMSE, PSD-NMSE, and FC-MSE by 6.5%, 16.3%, and 10.5%, respectively, relative to uniform sampling over both dimensions, and shows strong downstream performance across six in- and out-of-domain tasks. The taskonomy reveals asymmetric, target-dependent transfer, while exploratory sealed-test evaluation shows larger descriptive gains for BIP policies when higher-order route spaces are available than for matched random controls. Together, these findings support organizing fMRI pretraining and adaptation by measured learning relations rather than treating domains and tasks as independent flat sets.