# A Multi-modal Approach to Dysarthria Detection and Severity Assessment Using Speech and Text Information

arXiv: https://arxiv.org/abs/2412.16874 · HTML: https://mlsanity.com/paper/2412.16874/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 24 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Wav2vec-based Detection and Severity Level Classification of Dysarthria from Speech](https://mlsanity.com/paper/2309.14107/) (2023)
- [Linguistic-Acoustic Similarity Based Accent Shift for Accent Recognition](https://mlsanity.com/paper/2204.03398/) (2022)
- [Automatic Assessment of Dysarthria Using Audio-visual Vowel Graph Attention Network](https://mlsanity.com/paper/2405.03254/) (2024)

## Built on by (most cited first, 1 of 1)

- [Bridging the Perceptual-Statistical Gap in Dysarthria Assessment: Why Machine Learning Still Falls Short](https://mlsanity.com/paper/2510.22237/) (2025)

## Abstract (arXiv)

Automatic detection and severity assessment of dysarthria are crucial for delivering targeted therapeutic interventions to patients. While most existing research focuses primarily on speech modality, this study introduces a novel approach that leverages both speech and text modalities. By employing cross-attention mechanism, our method learns the acoustic and linguistic similarities between speech and text representations. This approach assesses specifically the pronunciation deviations across different severity levels, thereby enhancing the accuracy of dysarthric detection and severity assessment. All the experiments have been performed using UA-Speech dysarthric database. Improved accuracies of 99.53% and 93.20% in detection, and 98.12% and 51.97% for severity assessment have been achieved when speaker-dependent and speaker-independent, unseen and seen words settings are used. These findings suggest that by integrating text information, which provides a reference linguistic knowledge, a more robust framework has been developed for dysarthric detection and assessment, thereby potentially leading to more effective diagnoses.