# MultiMAE for Brain MRIs: Robustness to Missing Inputs Using Multi-Modal Masked Autoencoder

arXiv: https://arxiv.org/abs/2509.11442 · HTML: https://mlsanity.com/paper/2509.11442/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 14 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [MedViT: A Robust Vision Transformer for Generalized Medical Image Classification](https://mlsanity.com/paper/2302.09462/) (2023)
- [TransMed: Transformers Advance Multi-modal Medical Image Classification](https://mlsanity.com/paper/2103.05940/) (2021)
- [Self-supervised learning methods and applications in medical imaging analysis: A survey](https://mlsanity.com/paper/2109.08685/) (2021)
- [Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentation](https://mlsanity.com/paper/2203.05573/) (2022)
- [The University of California San Francisco Preoperative Diffuse Glioma MRI (UCSF-PDGM) Dataset](https://mlsanity.com/paper/2109.00356/) (2021)
- [The Brain Tumor Segmentation (BraTS-METS) Challenge 2023: Brain Metastasis Segmentation on Pre-treatment MRI](https://mlsanity.com/paper/2306.00838/) (2023)

## Built on by (most cited first, 1 of 1)

- [BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability](https://mlsanity.com/paper/2605.13059/) (2026)

## Abstract (arXiv)

Missing input sequences are common in medical imaging data, posing a challenge for deep learning models reliant on complete input data. In this work, inspired by MultiMAE [2], we develop a masked autoencoder (MAE) paradigm for multi-modal, multi-task learning in 3D medical imaging with brain MRIs. Our method treats each MRI sequence as a separate input modality, leveraging a late-fusion-style transformer encoder to integrate multi-sequence information (multi-modal) and individual decoder streams for each modality for multi-task reconstruction. This pretraining strategy guides the model to learn rich representations per modality while also equipping it to handle missing inputs through cross-sequence reasoning. The result is a flexible and generalizable encoder for brain MRIs that infers missing sequences from available inputs and can be adapted to various downstream applications. We demonstrate the performance and robustness of our method against an MAE-ViT baseline in downstream segmentation and classification tasks, showing absolute improvement of 10.1 overall Dice score and 0.46 MCC over the baselines with missing input sequences. Our experiments demonstrate the strength of this pretraining strategy. The implementation is made available.