# Unified Multimodal Model for Brain MRI Imputation and Understanding

arXiv: https://arxiv.org/abs/2606.16484 · HTML: https://mlsanity.com/paper/2606.16484/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models](https://mlsanity.com/paper/2411.04996/) (2024)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation](https://mlsanity.com/paper/2505.05472/) (2025)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [RaTEScore: A Metric for Radiology Report Generation](https://mlsanity.com/paper/2406.16845/) (2024)
- [Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision](https://mlsanity.com/paper/2508.05606/) (2025)
- [VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge](https://mlsanity.com/paper/2411.12915/) (2024)
- [Reconstruction Alignment Improves Unified Multimodal Models](https://mlsanity.com/paper/2509.07295/) (2025)
- [Brain Tumor Segmentation on MRI with Missing Modalities](https://mlsanity.com/paper/1904.07290/) (2019)
- [Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2407.05374/) (2024)
- [SimMLM: A Simple Framework for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2507.19264/) (2025)
- [UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis](https://mlsanity.com/paper/2510.15710/) (2025)
- [Regression is all you need for medical image translation](https://mlsanity.com/paper/2505.02048/) (2025)
- [Uni-COAL: A Unified Framework for Cross-Modality Synthesis and Super-Resolution of MR Images](https://mlsanity.com/paper/2311.08225/) (2023)

## Abstract (arXiv)

Multimodal large language models (MLLMs) hold great potential for medicine, as they inherit knowledge from LLM and allow multiple data modalities to be integrated, analysed and interpreted in natural language. However, the field of medical MLLMs is constrained by non-trivial challenges, notably the scarcity of high-quality training data and the frequent occurrence of missing data in the real-world clinical setting. Here, we propose a novel unified multimodal model, UniBrain, for brain magnetic resonance image (MRI) analysis. To address potential missing brain MRI modalities, we employ a unified training strategy to perform joint imaging modality imputation and brain image understanding. During training, an interleaved and description-enriched data flow is constructed to train the model in an autoregressive manner, enabling medical reasoning with generated multimodal data. A self-alignment strategy is introduced to leverage dense image embeddings to learn fine-grained anatomical features without requiring detailed image captions. Furthermore, we propose a dynamic hidden state mechanism to alleviate the exposure bias during long-context multimodal inference. Extensive experiments on multi-disease brain MRI dataset demonstrate that UniBrain achieves high performance for brain image imputation, understanding, and disease diagnosis under various extents of modality incompleteness.