# DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model

arXiv: https://arxiv.org/abs/2506.00350 · HTML: https://mlsanity.com/paper/2506.00350/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 31 May 2025.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 23 of 23)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2406.05370/) (2024)
- [SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models](https://mlsanity.com/paper/2406.02328/) (2024)
- [UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization](https://mlsanity.com/paper/2401.14664/) (2024)
- [Learning Explicit Prosody Models and Deep Speaker Embeddings for Atypical Voice Conversion](https://mlsanity.com/paper/2011.01678/) (2020)
- [Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy](https://mlsanity.com/paper/2403.16078/) (2024)
- [StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis](https://mlsanity.com/paper/2312.12181/) (2023)
- [CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction](https://mlsanity.com/paper/2406.08336/) (2024)
- [Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction](https://mlsanity.com/paper/2401.17796/) (2024)
- [ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training](https://mlsanity.com/paper/2501.04416/) (2025)

## Built on by (most cited first, 5 of 5)

- [Transcript-Free Flow-Matching Text-to-Speech via Speech Feature Conditioning](https://mlsanity.com/paper/2606.20266/) (2026)
- [An Evaluation Framework for Text-to-Speech Voice Reconstruction](https://mlsanity.com/paper/2606.21343/) (2026)
- [ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2511.06288/) (2025)
- [AP-GRPO: Anchor-Gated Phonetic Alignment with Policy Optimization for Pathological Speech Reconstruction](https://mlsanity.com/paper/2606.15540/) (2026)
- [Analyzing Speech Condition Effects in Dysarthric ASR: A Layer-wise Probing Study](https://mlsanity.com/paper/2608.01865/) (2026)

## Abstract (arXiv)

Dysarthric speech reconstruction (DSR) aims to convert dysarthric speech into comprehensible speech while maintaining the speaker's identity. Despite significant advancements, existing methods often struggle with low speech intelligibility and poor speaker similarity. In this study, we introduce a novel diffusion-based DSR system that leverages a latent diffusion model to enhance the quality of speech reconstruction. Our model comprises: (i) a speech content encoder for phoneme embedding restoration via pre-trained self-supervised learning (SSL) speech foundation models; (ii) a speaker identity encoder for speaker-aware identity preservation by in-context learning mechanism; (iii) a diffusion-based speech generator to reconstruct the speech based on the restored phoneme embedding and preserved speaker identity. Through evaluations on the widely-used UASpeech corpus, our proposed model shows notable enhancements in speech intelligibility and speaker similarity.