# Is a PET all you need? A multi-modal study for Alzheimer's disease using 3D CNNs

arXiv: https://arxiv.org/abs/2207.02094 · HTML: https://mlsanity.com/paper/2207.02094/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 8 of 8)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification](https://mlsanity.com/paper/1701.06643/) (2017)
- [Predicting Alzheimer's disease: a neuroimaging study with 3D convolutional neural networks](https://mlsanity.com/paper/1502.02506/) (2015)
- [Diagnosis of Alzheimer's Disease via Multi-modality 3D Convolutional Neural Network](https://mlsanity.com/paper/1902.09904/) (2019)
- [End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification](https://mlsanity.com/paper/1810.00523/) (2018)
- [Alzheimer's Disease Diagnostics by a Deeply Supervised Adaptable 3D Convolutional Network](https://mlsanity.com/paper/1607.00556/) (2016)

## Built on by (most cited first, 11 of 11)

- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [How You Split Matters: Data Leakage and Subject Characteristics Studies in Longitudinal Brain MRI Analysis](https://mlsanity.com/paper/2309.00350/) (2023)
- [Don't PANIC: Prototypical Additive Neural Network for Interpretable Classification of Alzheimer's Disease](https://mlsanity.com/paper/2303.07125/) (2023)
- [DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers Using MRI and PET](https://mlsanity.com/paper/2410.23219/) (2024)
- [RegBN: Batch Normalization of Multimodal Data with Regularization](https://mlsanity.com/paper/2310.00641/) (2023)
- [Deep Multimodal Fusion for Surgical Feedback Classification](https://mlsanity.com/paper/2312.03231/) (2023)
- [Attention-based 3D CNN with Multi-layer Features for Alzheimer's Disease Diagnosis using Brain Images](https://mlsanity.com/paper/2308.05655/) (2023)
- [Multi-Dimension-Embedding-Aware Modality Fusion Transformer for Psychiatric Disorder Clasification](https://mlsanity.com/paper/2310.02690/) (2023)
- [A Bi-Pyramid Multimodal Fusion Method for the Diagnosis of Bipolar Disorders](https://mlsanity.com/paper/2401.07571/) (2024)
- [Can Multitask Learning Enhance Model Explainability?](https://mlsanity.com/paper/2508.06966/) (2025)
- [Evidence Before Accuracy: A MRI-PET Fusion Network for Alzheimer Disease Classification with Causal Regional Validation](https://mlsanity.com/paper/2609.34520/) (2026)

## Abstract (arXiv)

Alzheimer's Disease (AD) is the most common form of dementia and often difficult to diagnose due to the multifactorial etiology of dementia. Recent works on neuroimaging-based computer-aided diagnosis with deep neural networks (DNNs) showed that fusing structural magnetic resonance images (sMRI) and fluorodeoxyglucose positron emission tomography (FDG-PET) leads to improved accuracy in a study population of healthy controls and subjects with AD. However, this result conflicts with the established clinical knowledge that FDG-PET better captures AD-specific pathologies than sMRI. Therefore, we propose a framework for the systematic evaluation of multi-modal DNNs and critically re-evaluate single- and multi-modal DNNs based on FDG-PET and sMRI for binary healthy vs. AD, and three-way healthy/mild cognitive impairment/AD classification. Our experiments demonstrate that a single-modality network using FDG-PET performs better than MRI (accuracy 0.91 vs 0.87) and does not show improvement when combined. This conforms with the established clinical knowledge on AD biomarkers, but raises questions about the true benefit of multi-modal DNNs. We argue that future work on multi-modal fusion should systematically assess the contribution of individual modalities following our proposed evaluation framework. Finally, we encourage the community to go beyond healthy vs. AD classification and focus on differential diagnosis of dementia, where fusing multi-modal image information conforms with a clinical need.