# Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech

arXiv: https://arxiv.org/abs/2506.01618 · HTML: https://mlsanity.com/paper/2506.01618/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Voice Conversion With Just Nearest Neighbors](https://mlsanity.com/paper/2305.18975/) (2023)
- [Training Data Augmentation for Dysarthric Automatic Speech Recognition by Text-to-Dysarthric-Speech Synthesis](https://mlsanity.com/paper/2406.08568/) (2024)
- [Synthesizing Dysarthric Speech Using Multi-talker TTS for Dysarthric Speech Recognition](https://mlsanity.com/paper/2201.11571/) (2022)
- [Rhythm Modeling for Voice Conversion](https://mlsanity.com/paper/2307.06040/) (2023)
- [Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR](https://mlsanity.com/paper/2501.10256/) (2025)

## Built on by (most cited first, 1 of 1)

- [Controllable Dysarthric Speech Synthesis with Patient-Specific Conditioning for Speaker-Diverse ASR Augmentation](https://mlsanity.com/paper/2602.08696/) (2026)

## Abstract (arXiv)

Automatic speech recognition (ASR) systems struggle with dysarthric speech due to high inter-speaker variability and slow speaking rates. To address this, we explore dysarthric-to-healthy speech conversion for improved ASR performance. Our approach extends the Rhythm and Voice (RnV) conversion framework by introducing a syllable-based rhythm modeling method suited for dysarthric speech. We assess its impact on ASR by training LF-MMI models and fine-tuning Whisper on converted speech. Experiments on the Torgo corpus reveal that LF-MMI achieves significant word error rate reductions, especially for more severe cases of dysarthria, while fine-tuning Whisper on converted data has minimal effect on its performance. These results highlight the potential of unsupervised rhythm and voice conversion for dysarthric ASR. Code available at: https://github.com/idiap/RnV