# Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today

arXiv: https://arxiv.org/abs/2306.01499 · HTML: https://mlsanity.com/paper/2306.01499/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations](https://mlsanity.com/paper/2303.18027/) (2023)
- [A Bibliometric Review of Large Language Models Research from 2017 to 2023](https://mlsanity.com/paper/2304.02020/) (2023)
- [Scalable Rule-Based Representation Learning for Interpretable Classification](https://mlsanity.com/paper/2109.15103/) (2021)
- [Large Language Models for Business Process Management: Opportunities and Challenges](https://mlsanity.com/paper/2304.04309/) (2023)

## Built on by (most cited first, 14 of 14)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection](https://mlsanity.com/paper/2402.13276/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [PhenoFlow: A Human-LLM Driven Visual Analytics System for Exploring Large and Complex Stroke Datasets](https://mlsanity.com/paper/2407.16329/) (2024)
- [Linguistic Features Extracted by GPT-4 Improve Alzheimer's Disease Detection based on Spontaneous Speech](https://mlsanity.com/paper/2412.15772/) (2024)
- [Evaluating Large Language Models for Public Health Classification and Extraction Tasks](https://mlsanity.com/paper/2405.14766/) (2024)
- [TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models](https://mlsanity.com/paper/2406.04941/) (2024)
- [Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences](https://mlsanity.com/paper/2502.20478/) (2025)
- [A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification](https://mlsanity.com/paper/2401.13887/) (2024)
- [Devising a Set of Compact and Explainable Spoken Language Feature for Screening Alzheimer's Disease](https://mlsanity.com/paper/2411.18922/) (2024)
- [Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance](https://mlsanity.com/paper/2508.13579/) (2025)
- [Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detection](https://mlsanity.com/paper/2506.03476/) (2025)
- [A Tutorial on Clinical Speech AI Development: From Data Collection to Model Validation](https://mlsanity.com/paper/2410.21640/) (2024)

## Abstract (arXiv)

Recent investigations show that large language models (LLMs), specifically GPT-4, not only have remarkable capabilities in common Natural Language Processing (NLP) tasks but also exhibit human-level performance on various professional and academic benchmarks. However, whether GPT-4 can be directly used in practical applications and replace traditional artificial intelligence (AI) tools in specialized domains requires further experimental validation. In this paper, we explore the potential of LLMs such as GPT-4 to outperform traditional AI tools in dementia diagnosis. Comprehensive comparisons between GPT-4 and traditional AI tools are conducted to examine their diagnostic accuracy in a clinical setting. Experimental results on two real clinical datasets show that, although LLMs like GPT-4 demonstrate potential for future advancements in dementia diagnosis, they currently do not surpass the performance of traditional AI tools. The interpretability and faithfulness of GPT-4 are also evaluated by comparison with real doctors. We discuss the limitations of GPT-4 in its current state and propose future research directions to enhance GPT-4 in dementia diagnosis.