# Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens

arXiv: https://arxiv.org/abs/2509.24693 · HTML: https://mlsanity.com/paper/2509.24693/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2025.

## Built on by (most cited first, 15 of 15)

- [FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI](https://mlsanity.com/paper/2606.11500/) (2026)
- [Scaling Vision Transformers for Functional MRI with Flat Maps](https://mlsanity.com/paper/2510.13768/) (2025)
- [BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics](https://mlsanity.com/paper/2606.17742/) (2026)
- [Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG](https://mlsanity.com/paper/2602.23410/) (2026)
- [The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail](https://mlsanity.com/paper/2606.04010/) (2026)
- [BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructure](https://mlsanity.com/paper/2607.00573/) (2026)
- [Multimodal Visual Surrogate Compression for Alzheimer's Disease Classification](https://mlsanity.com/paper/2601.21673/) (2026)
- [BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction](https://mlsanity.com/paper/2602.04512/) (2026)
- [One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models](https://mlsanity.com/paper/2602.21522/) (2026)
- [Real-time Reconstruction of Human Visual Perception from fMRI](https://mlsanity.com/paper/2607.22753/) (2026)
- [BrainNorm: A Foundation Model that knows Normal via Semantic Atlas Pretraining](https://mlsanity.com/paper/2608.17521/) (2026)
- [A Scaling Study for fMRI Foundation Models](https://mlsanity.com/paper/2609.27232/) (2026)
- [FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps](https://mlsanity.com/paper/2609.31204/) (2026)
- [Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction](https://mlsanity.com/paper/2609.34167/) (2026)
- [Flattening the Connectome Spectrum: A Spectral Filter for FC Induces a Pretraining Target for fMRI Encoders](https://mlsanity.com/paper/2609.37642/) (2026)

## Abstract (arXiv)

We present Brain Harmony (BrainHarmonix), the first multimodal brain foundation model that unifies structural morphology and functional dynamics into compact 1D token representations. The model was pretrained on two of the largest neuroimaging datasets to date, encompassing 64,594 T1-weighted structural MRI 3D volumes (~ 14 million images) and 70,933 functional MRI (fMRI) time series. BrainHarmonix is grounded in two foundational neuroscience principles: structure complements function - structural and functional modalities offer distinct yet synergistic insights into brain organization; function follows structure - brain functional dynamics are shaped by cortical morphology. The modular pretraining process involves single-modality training with geometric pre-alignment followed by modality fusion through shared brain hub tokens. Notably, our dynamics encoder uniquely handles fMRI time series with heterogeneous repetition times (TRs), addressing a major limitation in existing models. BrainHarmonix is also the first to deeply compress high-dimensional neuroimaging signals into unified, continuous 1D tokens, forming a compact latent space of the human brain. BrainHarmonix achieves strong generalization across diverse downstream tasks, including neurodevelopmental and neurodegenerative disorder classification and cognition prediction - consistently outperforming previous approaches. Our models - pretrained on 8 H100 GPUs - aim to catalyze a new era of AI-driven neuroscience powered by large-scale multimodal neuroimaging.