# Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data

arXiv: https://arxiv.org/abs/2210.00255 · HTML: https://mlsanity.com/paper/2210.00255/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 59 references that are in the map.
- First posted on arXiv 1 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation](https://mlsanity.com/paper/1904.07773/) (2019)
- [Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification](https://mlsanity.com/paper/1701.06643/) (2017)
- [Visual Explanations From Deep 3D Convolutional Neural Networks for Alzheimer's Disease Classification](https://mlsanity.com/paper/1803.02544/) (2018)
- [M3d-CAM: A PyTorch library to generate 3D data attention maps for medical deep learning](https://mlsanity.com/paper/2007.00453/) (2020)
- [MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data](https://mlsanity.com/paper/2104.03618/) (2021)

## Built on by (most cited first, 17 of 17)

- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications](https://mlsanity.com/paper/2411.03782/) (2024)
- [FRAMM: Fair Ranking with Missing Modalities for Clinical Trial Site Selection](https://mlsanity.com/paper/2305.19407/) (2023)
- [Toward Robust Early Detection of Alzheimer's Disease via an Integrated Multimodal Learning Approach](https://mlsanity.com/paper/2408.16343/) (2024)
- [MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems](https://mlsanity.com/paper/2405.01658/) (2024)
- [MARIA: a Multimodal Transformer Model for Incomplete Healthcare Data](https://mlsanity.com/paper/2412.14810/) (2024)
- [GFE-Mamba: Mamba-based AD Multi-modal Progression Assessment via Generative Feature Extraction from MCI](https://mlsanity.com/paper/2407.15719/) (2024)
- [CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation](https://mlsanity.com/paper/2309.03320/) (2023)
- [Exploiting Field Dependencies for Learning on Categorical Data](https://mlsanity.com/paper/2307.09321/) (2023)
- [Looking deeper into interpretable deep learning in neuroimaging: a comprehensive survey](https://mlsanity.com/paper/2307.09615/) (2023)
- [Not Only Grey Matter: OmniBrain for Robust Multimodal Classification of Alzheimer's Disease](https://mlsanity.com/paper/2507.20872/) (2025)
- [AD-CARE: A Guideline-grounded, Modality-agnostic LLM Agent for Real-world Alzheimer's Disease Diagnosis with Multi-cohort Assessment, Fairness Analysis, and Reader Study](https://mlsanity.com/paper/2603.25322/) (2026)
- [EffNetViTLoRA: An Efficient Hybrid Deep Learning Approach for Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2508.19349/) (2025)
- [PGAD: Prototype-Guided Adaptive Distillation for Multi-Modal Learning in AD Diagnosis](https://mlsanity.com/paper/2503.04836/) (2025)
- [Sparse Multi-Modal Transformer with Masking for Alzheimer's Disease Classification](https://mlsanity.com/paper/2512.14491/) (2025)
- [MRC-GAT: A Meta-Relational Copula-Based Graph Attention Network for Interpretable Multimodal Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2602.15740/) (2026)
- [Multiple Inputs and Mixwd data for Alzheimer's Disease Classification Based on 3D Vision Transformer](https://mlsanity.com/paper/2603.00545/) (2026)

## Abstract (arXiv)

Accurate medical classification requires a large number of multi-modal data, and in many cases, different feature types. Previous studies have shown promising results when using multi-modal data, outperforming single-modality models when classifying diseases such as Alzheimer's Disease (AD). However, those models are usually not flexible enough to handle missing modalities. Currently, the most common workaround is discarding samples with missing modalities which leads to considerable data under-utilization. Adding to the fact that labeled medical images are already scarce, the performance of data-driven methods like deep learning can be severely hampered. Therefore, a multi-modal method that can handle missing data in various clinical settings is highly desirable. In this paper, we present Multi-Modal Mixing Transformer (3MAT), a disease classification transformer that not only leverages multi-modal data but also handles missing data scenarios. In this work, we test 3MT for AD and Cognitively normal (CN) classification and mild cognitive impairment (MCI) conversion prediction to progressive MCI (pMCI) or stable MCI (sMCI) using clinical and neuroimaging data. The model uses a novel Cascaded Modality Transformer architecture with cross-attention to incorporate multi-modal information for more informed predictions. We propose a novel modality dropout mechanism to ensure an unprecedented level of modality independence and robustness to handle missing data scenarios. The result is a versatile network that enables the mixing of arbitrary numbers of modalities with different feature types and also ensures full data utilization missing data scenarios. The model is trained and evaluated on the ADNI dataset with the SOTRA performance and further evaluated with the AIBL dataset with missing data.