# DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers Using MRI and PET

arXiv: https://arxiv.org/abs/2410.23219 · HTML: https://mlsanity.com/paper/2410.23219/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 44 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [3D TransUNet: Advancing Medical Image Segmentation through Vision Transformers](https://mlsanity.com/paper/2310.07781/) (2023)
- [Diagnosis of Alzheimer's Disease via Multi-modality 3D Convolutional Neural Network](https://mlsanity.com/paper/1902.09904/) (2019)
- [Metadata Normalization](https://mlsanity.com/paper/2104.09052/) (2021)
- [Is a PET all you need? A multi-modal study for Alzheimer's disease using 3D CNNs](https://mlsanity.com/paper/2207.02094/) (2022)
- [Multimodal and Multiscale Deep Neural Networks for the Early Diagnosis of Alzheimer's Disease using structural MR and FDG-PET images](https://mlsanity.com/paper/1710.04782/) (2017)
- [Interpretable differential diagnosis for Alzheimer's disease and Frontotemporal dementia](https://mlsanity.com/paper/2206.07417/) (2022)
- [Multiple Instance Neuroimage Transformer](https://mlsanity.com/paper/2208.09567/) (2022)
- [RegBN: Batch Normalization of Multimodal Data with Regularization](https://mlsanity.com/paper/2310.00641/) (2023)
- [From Barlow Twins to Triplet Training: Differentiating Dementia with Limited Data](https://mlsanity.com/paper/2404.06253/) (2024)

## Built on by (most cited first, 6 of 6)

- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [Dementia-Agents: A Multi-Modal Multi-Agent System for Dementia Staging and Phenotyping](https://mlsanity.com/paper/2606.21168/) (2026)
- [AT-Attn: Temporal-Aware Cross-Attention for Longitudinal Multimodal Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2607.07091/) (2026)
- [Modality Contribution Score - A Per-Patient Framework for Quantifying the Relative Diagnostic Contribution of Structural MRI and Amyloid PET in Alzheimer's Disease](https://mlsanity.com/paper/2608.24931/) (2026)
- [PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology](https://mlsanity.com/paper/2608.25970/) (2026)
- [Evidence Before Accuracy: A MRI-PET Fusion Network for Alzheimer Disease Classification with Causal Regional Validation](https://mlsanity.com/paper/2609.34520/) (2026)

## Abstract (arXiv)

Diagnosing dementia, particularly for Alzheimer's Disease (AD) and frontotemporal dementia (FTD), is complex due to overlapping symptoms. While magnetic resonance imaging (MRI) and positron emission tomography (PET) data are critical for the diagnosis, integrating these modalities in deep learning faces challenges, often resulting in suboptimal performance compared to using single modalities. Moreover, the potential of multi-modal approaches in differential diagnosis, which holds significant clinical importance, remains largely unexplored. We propose a novel framework, DiaMond, to address these issues with vision Transformers to effectively integrate MRI and PET. DiaMond is equipped with self-attention and a novel bi-attention mechanism that synergistically combine MRI and PET, alongside a multi-modal normalization to reduce redundant dependency, thereby boosting the performance. DiaMond significantly outperforms existing multi-modal methods across various datasets, achieving a balanced accuracy of 92.4% in AD diagnosis, 65.2% for AD-MCI-CN classification, and 76.5% in differential diagnosis of AD and FTD. We also validated the robustness of DiaMond in a comprehensive ablation study. The code is available at https://github.com/ai-med/DiaMond.