# HoloDx: Knowledge- and Data-Driven Multimodal Diagnosis of Alzheimer's Disease

arXiv: https://arxiv.org/abs/2504.19075 · HTML: https://mlsanity.com/paper/2504.19075/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 27 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale](https://mlsanity.com/paper/2406.19280/) (2024)
- [Automated brain extraction of multi-sequence MRI using artificial neural networks](https://mlsanity.com/paper/1901.11341/) (2019)
- [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](https://mlsanity.com/paper/2306.00864/) (2023)
- [With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning](https://mlsanity.com/paper/2308.12383/) (2023)
- [Heterogeneous Graph Learning for Multi-modal Medical Data Analysis](https://mlsanity.com/paper/2211.15158/) (2022)
- [AliFuse: Aligning and Fusing Multi-modal Medical Data for Computer-Aided Diagnosis](https://mlsanity.com/paper/2401.01074/) (2024)
- [Label Dependent Attention Model for Disease Risk Prediction Using Multimodal Electronic Health Records](https://mlsanity.com/paper/2201.06779/) (2022)
- [Label-dependent and event-guided interpretable disease risk prediction using EHRs](https://mlsanity.com/paper/2201.06783/) (2022)

## Built on by (most cited first, 3 of 3)

- [EMAD: Evidence-Centric Grounded Multimodal Diagnosis for Alzheimer's Disease](https://mlsanity.com/paper/2602.19178/) (2026)
- [A Quantitative Analysis of Multimodal Biomarkers in Alzheimer's Disease](https://mlsanity.com/paper/2606.17867/) (2026)
- [Anatomy-aware Fine-grained Multimodal Fusion for Laryngopharyngeal Cancer T-Staging Prediction Using CT and Radiology Report](https://mlsanity.com/paper/2610.06837/) (2026)

## Abstract (arXiv)

Accurate diagnosis of Alzheimer's disease (AD) requires effectively integrating multimodal data and clinical expertise. However, existing methods often struggle to fully utilize multimodal information and lack structured mechanisms to incorporate dynamic domain knowledge. To address these limitations, we propose HoloDx, a knowledge- and data-driven framework that enhances AD diagnosis by aligning domain knowledge with multimodal clinical data. HoloDx incorporates a knowledge injection module with a knowledge-aware gated cross-attention, allowing the model to dynamically integrate domain-specific insights from both large language models (LLMs) and clinical expertise. Also, a memory injection module with a designed prototypical memory attention enables the model to retain and retrieve subject-specific information, ensuring consistency in decision-making. By jointly leveraging these mechanisms, HoloDx enhances interpretability, improves robustness, and effectively aligns prior knowledge with current subject data. Evaluations on five AD datasets demonstrate that HoloDx outperforms state-of-the-art methods, achieving superior diagnostic accuracy and strong generalization across diverse cohorts. The source code will be released upon publication acceptance.