# Developing Conversational Speech Systems for Robots to Detect Speech Biomarkers of Cognition in People Living with Dementia

arXiv: https://arxiv.org/abs/2502.10896 · HTML: https://mlsanity.com/paper/2502.10896/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 96 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Builds on (most cited first, 7 of 7)

- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models](https://mlsanity.com/paper/1804.06440/) (2018)
- [Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech](https://mlsanity.com/paper/2106.09668/) (2021)
- [Learning to detect dysarthria from raw speech](https://mlsanity.com/paper/1811.11101/) (2018)
- [A Method for Analysis of Patient Speech in Dialogue for Dementia Detection](https://mlsanity.com/paper/1811.09919/) (2018)
- [Identification of Dementia Using Audio Biomarkers](https://mlsanity.com/paper/2002.12788/) (2020)

## Built on by (most cited first, 1 of 1)

- [Designing and Evaluating a Conversational Agent for Early Diagnosis of Alzheimer's Disease and Related Dementias](https://mlsanity.com/paper/2509.11478/) (2025)

## Abstract (arXiv)

This study presents the development and testing of a conversational speech system designed for robots to detect speech biomarkers indicative of cognitive impairments in people living with dementia (PLwD). The system integrates a backend Python WebSocket server and a central core module with a large language model (LLM) fine-tuned for dementia to process user input and generate robotic conversation responses in real-time in less than 1.5 seconds. The frontend user interface, a Progressive Web App (PWA), displays information and biomarker score graphs on a smartphone in real-time to human users (PLwD, caregivers, clinicians). Six speech biomarkers based on the existing literature - Altered Grammar, Pragmatic Impairments, Anomia, Disrupted Turn-Taking, Slurred Pronunciation, and Prosody Changes - were developed for the robot conversation system using two datasets, one that included conversations of PLwD with a human clinician (DementiaBank dataset) and one that included conversations of PLwD with a robot (Indiana dataset). We also created a composite speech biomarker that combined all six individual biomarkers into a single score. The speech system's performance was first evaluated on the DementiaBank dataset showing moderate correlation with MMSE scores, with the composite biomarker score outperforming individual biomarkers. Analysis of the Indiana dataset revealed higher and more variable biomarker scores, suggesting potential differences due to study populations (e.g. severity of dementia) and the conversational scenario (human-robot conversations are different from human-human). The findings underscore the need for further research on the impact of conversational scenarios on speech biomarkers and the potential clinical applications of robotic speech systems.