# Large language models improve Alzheimer's disease diagnosis using multi-modality data

arXiv: https://arxiv.org/abs/2305.19280 · HTML: https://mlsanity.com/paper/2305.19280/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://mlsanity.com/paper/2302.07257/) (2023)
- [Feature robustness and sex differences in medical imaging: a case study in MRI-based Alzheimer's disease detection](https://mlsanity.com/paper/2204.01737/) (2022)
- [DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection](https://mlsanity.com/paper/2305.12519/) (2023)
- [Gene Set Summarization using Large Language Models](https://mlsanity.com/paper/2305.13338/) (2023)
- [Multi-Modal Hypergraph Diffusion Network with Dual Prior for Alzheimer Classification](https://mlsanity.com/paper/2204.02399/) (2022)
- [Multi-scale Graph-based Grading for Alzheimer's Disease Prediction](https://mlsanity.com/paper/1907.06625/) (2019)

## Built on by (most cited first, 9 of 9)

- [DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature](https://mlsanity.com/paper/2405.04819/) (2024)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [LLMs-Healthcare : Current Applications and Challenges of Large Language Models in various Medical Specialties](https://mlsanity.com/paper/2311.12882/) (2023)
- [MedOrch: Medical Diagnosis with Tool-Augmented Reasoning Agents for Flexible Extensibility](https://mlsanity.com/paper/2506.00235/) (2025)
- [Large Language Models for Bioinformatics](https://mlsanity.com/paper/2501.06271/) (2025)
- [ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Study](https://mlsanity.com/paper/2401.06513/) (2024)
- [Leveraging Multimodal Models for Enhanced Neuroimaging Diagnostics in Alzheimer's Disease](https://mlsanity.com/paper/2411.07871/) (2024)
- [Cross-Attention Fusion of MRI and Jacobian Maps for Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2503.00586/) (2025)
- [Speech-based Psychological Crisis Assessment using LLMs](https://mlsanity.com/paper/2605.10027/) (2026)

## Abstract (arXiv)

In diagnosing challenging conditions such as Alzheimer's disease (AD), imaging is an important reference. Non-imaging patient data such as patient information, genetic data, medication information, cognitive and memory tests also play a very important role in diagnosis. Effect. However, limited by the ability of artificial intelligence models to mine such information, most of the existing models only use multi-modal image data, and cannot make full use of non-image data. We use a currently very popular pre-trained large language model (LLM) to enhance the model's ability to utilize non-image data, and achieved SOTA results on the ADNI dataset.