# Differentiable and Severity-invariant Discrete Tokens for Dysarthric Speech Recognition

arXiv: https://arxiv.org/abs/2609.16855 · HTML: https://mlsanity.com/paper/2609.16855/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in ASR evaluation, bias and atypical speech, one of 11 directions in Speech and audio language models.
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Placed in

- Speech and audio language models / [ASR evaluation, bias and atypical speech](https://mlsanity.com/speech/asr-evaluation-bias-and-atypical-speech/)

## Builds on (most cited first, 30 of 32)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study](https://mlsanity.com/paper/2309.15800/) (2023)
- [VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature](https://mlsanity.com/paper/2204.00768/) (2022)
- [UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding](https://mlsanity.com/paper/2306.07547/) (2023)
- [Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS](https://mlsanity.com/paper/2309.07377/) (2023)
- [Investigation of Data Augmentation Techniques for Disordered Speech Recognition](https://mlsanity.com/paper/2201.05562/) (2022)
- [Speaker adaptation for Wav2vec2 based dysarthric ASR](https://mlsanity.com/paper/2204.00770/) (2022)
- [Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2202.10290/) (2022)
- [Adversarial Data Augmentation for Disordered Speech Recognition](https://mlsanity.com/paper/2108.00899/) (2021)
- [Cross-lingual Self-Supervised Speech Representations for Improved Dysarthric Speech Recognition](https://mlsanity.com/paper/2204.01670/) (2022)
- [Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2302.14564/) (2023)
- [Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation](https://mlsanity.com/paper/2401.00662/) (2024)
- [Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2407.13782/) (2024)
- [DuTa-VC: A Duration-aware Typical-to-atypical Voice Conversion Approach with Diffusion Probabilistic Model](https://mlsanity.com/paper/2306.10588/) (2023)
- [Deep clustering with concrete k-means](https://mlsanity.com/paper/1910.08031/) (2019)
- [Adversarial Data Augmentation Using VAE-GAN for Disordered Speech Recognition](https://mlsanity.com/paper/2211.01646/) (2022)
- [Use of Speech Impairment Severity for Dysarthric Speech Recognition](https://mlsanity.com/paper/2305.10659/) (2023)
- [Estimating the Completeness of Discrete Speech Units](https://mlsanity.com/paper/2409.06109/) (2024)
- [Two-pass Decoding and Cross-adaptation Based System Combination of End-to-end Conformer and Hybrid TDNN ASR Systems](https://mlsanity.com/paper/2206.11596/) (2022)
- [Children's Speech Recognition through Discrete Token Enhancement](https://mlsanity.com/paper/2406.13431/) (2024)
- [Differentiable K-means for Fully-optimized Discrete Token-based ASR](https://mlsanity.com/paper/2505.16207/) (2025)
- [Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion](https://mlsanity.com/paper/2505.24291/) (2025)
- [Accent Normalization Using Self-Supervised Discrete Tokens with Non-Parallel Data](https://mlsanity.com/paper/2507.17735/) (2025)
- [Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR](https://mlsanity.com/paper/2409.08797/) (2024)
- [Phone-purity Guided Discrete Tokens for Dysarthric Speech Recognition](https://mlsanity.com/paper/2501.04379/) (2025)

## Abstract (arXiv)

This paper proposes novel differentiable and severity-invariant (DSI) discrete token approaches that are not only tightly integrated with downstream dysarthric speech recognition tasks, but also minimise discrete token diversity across speech impairment severity groups. Experiments conducted on the UASpeech and TORGO corpora suggest that Conformer models trained using the DSI tokens outperform the comparable baseline HuBERT discrete/continuous features by statistically significant WER reductions of 2.22%/0.78% absolute (9.14%/3.41% relative) and 1.78%/1.06% absolute (18.43%/11.86% relative) on the two tasks, respectively. After system combination, the lowest WERs of 18.90% and 6.38% were obtained on UASpeech and TORGO. Phoneme-specific T-SNE visualizations show that severity-invariant regularization reduces severity-dependent variation by producing greater overlap and less distinct boundaries among severity-group distributions.