# Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition

arXiv: https://arxiv.org/abs/2505.12991 · HTML: https://mlsanity.com/paper/2505.12991/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2025.

## Built on by (most cited first, 6 of 6)

- [When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition](https://mlsanity.com/paper/2605.02782/) (2026)
- [FiLM-Based Speaker Conditioning of a SpeechLLM for Pathological Speech Recognition](https://mlsanity.com/paper/2606.06211/) (2026)
- [An Evaluation Framework for Text-to-Speech Voice Reconstruction](https://mlsanity.com/paper/2606.21343/) (2026)
- [DARS: Dysarthria-Aware Rhythm-Style Synthesis for ASR Enhancement](https://mlsanity.com/paper/2603.01369/) (2026)
- [Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning](https://mlsanity.com/paper/2606.19823/) (2026)
- [Choosing a PEFT Variant for Per-Patient Dysarthric ASR: A Single-Speaker Case Study on Two ASR Bases](https://mlsanity.com/paper/2609.02735/) (2026)

## Abstract (arXiv)

In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ~23% and ~22%, respectively. Further improvements (~5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ~7% relative WER improvement over personalized fine-tuning alone.