# Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis

arXiv: https://arxiv.org/abs/2206.08826 · HTML: https://mlsanity.com/paper/2206.08826/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 29 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 2 of 2)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)

## Built on by (most cited first, 24 of 24)

- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications](https://mlsanity.com/paper/2411.03782/) (2024)
- [Multimodal Contrastive Learning and Tabular Attention for Automated Alzheimer's Disease Prediction](https://mlsanity.com/paper/2308.15469/) (2023)
- [One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Data](https://mlsanity.com/paper/2307.05435/) (2023)
- [FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Prediction](https://mlsanity.com/paper/2406.11928/) (2024)
- [NeuroMoE: A Transformer-Based Mixture-of-Experts Framework for Multi-Modal Neurological Disorder Classification](https://mlsanity.com/paper/2506.14970/) (2025)
- [PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset](https://mlsanity.com/paper/2507.03165/) (2025)
- [Not Only Grey Matter: OmniBrain for Robust Multimodal Classification of Alzheimer's Disease](https://mlsanity.com/paper/2507.20872/) (2025)
- [R-GenIMA: Integrating Neuroimaging and Genetics with Interpretable Multimodal AI for Alzheimer's Disease Progression](https://mlsanity.com/paper/2512.18986/) (2025)
- [HiMAL: A Multimodal Hierarchical Multi-task Auxiliary Learning framework for predicting and explaining Alzheimer disease progression](https://mlsanity.com/paper/2404.03208/) (2024)
- [MVKTrans: Multi-View Knowledge Transfer for Robust Multiomics Classification](https://mlsanity.com/paper/2411.08703/) (2024)
- [EffNetViTLoRA: An Efficient Hybrid Deep Learning Approach for Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2508.19349/) (2025)
- [Multimodal Neuroimaging Attention-Based architecture for Cognitive Decline Prediction](https://mlsanity.com/paper/2401.06777/) (2023)
- [CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings](https://mlsanity.com/paper/2501.18891/) (2025)
- [NeuroSymAD: A Neuro-Symbolic Framework for Interpretable Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2503.00510/) (2025)
- [Challenges and proposed solutions in modeling multimodal medical data: A systematic review](https://mlsanity.com/paper/2505.06945/) (2025)
- [Multi-omic Prognosis of Alzheimer's Disease with Asymmetric Cross-Modal Cross-Attention Network](https://mlsanity.com/paper/2507.08855/) (2025)
- [Sparse Multi-Modal Transformer with Masking for Alzheimer's Disease Classification](https://mlsanity.com/paper/2512.14491/) (2025)
- [Cross-modal ultra-scale learning with tri-modalities of renal biopsy images for glomerular multi-disease auxiliary diagnosis](https://mlsanity.com/paper/2512.15171/) (2025)
- [Causal Transfer in Medical Image Analysis](https://mlsanity.com/paper/2603.24388/) (2026)
- [Alzheimer's Disease Diagnosis using a Multimodal Approach with 3D MRI and PET](https://mlsanity.com/paper/2606.20037/) (2026)
- [Decoding Phenotypes: A Framework for Fusing Genomic Language Models and Neuroimaging](https://mlsanity.com/paper/2608.08926/) (2026)
- [Data-driven techniques for translational neuroscience and personalized neuro-health](https://mlsanity.com/paper/2608.13749/) (2026)
- [A Multimodal Explainable Deep Learning Framework for Alzheimer's Disease Diagnosis using 3D Magnetic Resonance Imaging and Clinical Data](https://mlsanity.com/paper/2609.12410/) (2026)

## Abstract (arXiv)

Alzheimer's Disease (AD) is the most common neurodegenerative disorder with one of the most complex pathogeneses, making effective and clinically actionable decision support difficult. The objective of this study was to develop a novel multimodal deep learning framework to aid medical professionals in AD diagnosis. We present a Multimodal Alzheimer's Disease Diagnosis framework (MADDi) to accurately detect the presence of AD and mild cognitive impairment (MCI) from imaging, genetic, and clinical data. MADDi is novel in that we use cross-modal attention, which captures interactions between modalities - a method not previously explored in this domain. We perform multi-class classification, a challenging task considering the strong similarities between MCI and AD. We compare with previous state-of-the-art models, evaluate the importance of attention, and examine the contribution of each modality to the model's performance. MADDi classifies MCI, AD, and controls with 96.88% accuracy on a held-out test set. When examining the contribution of different attention schemes, we found that the combination of cross-modal attention with self-attention performed the best, and no attention layers in the model performed the worst, with a 7.9% difference in F1-Scores. Our experiments underlined the importance of structured clinical data to help machine learning models contextualize and interpret the remaining modalities. Extensive ablation studies showed that any multimodal mixture of input features without access to structured clinical information suffered marked performance losses. This study demonstrates the merit of combining multiple input modalities via cross-modal attention to deliver highly accurate AD diagnostic decision support.