# Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2607.21496 · HTML: https://mlsanity.com/paper/2607.21496/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 58 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [WavLLM: Towards Robust and Adaptive Speech Large Language Model](https://mlsanity.com/paper/2404.00656/) (2024)
- [Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge](https://mlsanity.com/paper/2004.06833/) (2020)
- [Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS](https://mlsanity.com/paper/1912.06059/) (2019)
- [Detecting cognitive decline using speech only: The ADReSSo Challenge](https://mlsanity.com/paper/2104.09356/) (2021)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [A Survey on Speech Large Language Models for Understanding](https://mlsanity.com/paper/2410.18908/) (2024)
- [Artificial Intelligence, speech and language processing approaches to monitoring Alzheimer's Disease: a systematic review](https://mlsanity.com/paper/2010.06047/) (2020)
- [Exploiting Multi-Modal Features From Pre-trained Networks for Alzheimer's Dementia Recognition](https://mlsanity.com/paper/2009.04070/) (2020)
- [Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech](https://mlsanity.com/paper/2106.09668/) (2021)
- [Connected Speech-Based Cognitive Assessment in Chinese and English](https://mlsanity.com/paper/2406.10272/) (2024)
- [LLMCARE: early detection of cognitive impairment via transformer models enhanced by LLM-generated synthetic data](https://mlsanity.com/paper/2508.10027/) (2025)
- [A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection](https://mlsanity.com/paper/2410.19898/) (2024)
- [Cognitive Insights Across Languages: Enhancing Multimodal Interview Analysis](https://mlsanity.com/paper/2406.07542/) (2024)
- [Zero-Shot Cognitive Impairment Detection from Speech Using AudioLLM](https://mlsanity.com/paper/2506.17351/) (2025)

## Abstract (arXiv)

Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.