# Learning Sparse Latent Predictive Foundation Model for Multimodal Neuroimaging

arXiv: https://arxiv.org/abs/2606.14957 · HTML: https://mlsanity.com/paper/2606.14957/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 91 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [MIRAGE: The Illusion of Visual Understanding](https://mlsanity.com/paper/2603.21687/) (2026)
- [What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?](https://mlsanity.com/paper/2512.24497/) (2025)
- [Beyond Language Modeling: An Exploration of Multimodal Pretraining](https://mlsanity.com/paper/2603.03276/) (2026)
- [Causal-JEPA: Learning World Models through Object-Level Latent Masking](https://mlsanity.com/paper/2602.11389/) (2026)
- [EchoJEPA: A Latent Predictive Foundation Model for Echocardiography](https://mlsanity.com/paper/2602.02603/) (2026)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)

## Built on by (most cited first, 4 of 4)

- [RegionFM: Interpretable Region-Based Brain MRI Classification Using Foundation Model Embeddings](https://mlsanity.com/paper/2607.16325/) (2026)
- [BrainNorm: A Foundation Model that knows Normal via Semantic Atlas Pretraining](https://mlsanity.com/paper/2608.17521/) (2026)
- [Evaluating the Generalization of Neuroimaging Foundation Models on African Brain MRI](https://mlsanity.com/paper/2609.23983/) (2026)
- [The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging](https://mlsanity.com/paper/2610.13237/) (2026)

## Abstract (arXiv)

Brain MRIs are routinely acquired as multiple complementary sequences with unique contrast weighting, including T1-weighed imaging (T1w) anatomic and fluid-sensitive T2-weighted (T2w) contrasts. However, methods for learning unified representations across the multitude of MRI contrast mechanisms at health-system scale are lacking. In this study, we introduce Neuro-JEPA, a sparse multimodal neuroimaging foundation model that combines a latent predictive objective with a Mixture-of-Experts architecture to encode brain MRI across core T1w, T2w, and fluid-suppressed FLAIR imaging (FLAIR). We further provide a systematic methodological study of architectural, masking, objective, and sparsity design choices beneficial for robust neuroimaging multimodal representation learning. Neuro-JEPA was pretrained on 1,551,862 scans from 428,647 studies after modality-specific preprocessing with data curation across three core structural brain MRI sequences. We evaluated the learned representations across clinical and research settings, including 25 tasks from three health systems: NYU Langone, NYU Long Island, and Massachusetts General Hospital, and 22 tasks from 12 public datasets, covering unimodal, multimodal and cross-domain evaluation configurations. Across these benchmarks, existing neuroimaging foundation models showed inconsistent gains over a simple convolutional neural network (CNN) baseline, whereas Neuro-JEPA achieved stronger and more consistent performance across all evaluated settings. These results establish a scalable methodological framework for multimodal neuroimaging representation learning and highlight the need for foundation model evaluation protocols that include simple baselines, clinically heterogeneous cohorts and controlled multimodal comparisons.