# DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformers

arXiv: https://arxiv.org/abs/2504.09381 · HTML: https://mlsanity.com/paper/2504.09381/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 13 Apr 2025.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [Long-form music generation with latent diffusion](https://mlsanity.com/paper/2404.10301/) (2024)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2406.06185/) (2024)
- [SELM: Speech Enhancement Using Discrete Tokens and Language Models](https://mlsanity.com/paper/2312.09747/) (2023)
- [MaskSR: Masked Language Model for Full-band Speech Restoration](https://mlsanity.com/paper/2406.02092/) (2024)
- [Universal Score-based Speech Enhancement with High Content Preservation](https://mlsanity.com/paper/2406.12194/) (2024)
- [Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations](https://mlsanity.com/paper/2303.01664/) (2023)
- [Structure from Silence: Learning Scene Structure from Ambient Sound](https://mlsanity.com/paper/2111.05846/) (2021)
- [A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysis](https://mlsanity.com/paper/2011.00030/) (2020)
- [Low-latency Speech Enhancement via Speech Token Generation](https://mlsanity.com/paper/2310.08981/) (2023)
- [High-Resolution Speech Restoration with Latent Diffusion Model](https://mlsanity.com/paper/2409.11145/) (2024)

## Built on by (most cited first, 6 of 6)

- [AUDDT: A Unified Benchmark Toolkit for Audio and Speech Deepfake Detectors](https://mlsanity.com/paper/2509.21597/) (2025)
- [Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration](https://mlsanity.com/paper/2505.04457/) (2025)
- [Geneses: Unified Generative Speech Enhancement and Separation](https://mlsanity.com/paper/2601.18456/) (2026)
- [Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberation](https://mlsanity.com/paper/2505.16911/) (2025)
- [Absorbing Discrete Diffusion for Speech Enhancement](https://mlsanity.com/paper/2602.22417/) (2026)
- [A Cold Diffusion Approach for Percussive Dereverberation](https://mlsanity.com/paper/2605.10256/) (2026)

## Abstract (arXiv)

Real-world speech recordings suffer from degradations such as background noise and reverberation. Speech enhancement aims to mitigate these issues by generating clean high-fidelity signals. While recent generative approaches for speech enhancement have shown promising results, they still face two major challenges: (1) content hallucination, where plausible phonemes generated differ from the original utterance; and (2) inconsistency, failing to preserve speaker's identity and paralinguistic features from the input speech. In this work, we introduce DiTSE (Diffusion Transformer for Speech Enhancement), which addresses quality issues of degraded speech in full bandwidth. Our approach employs a latent diffusion transformer model together with robust conditioning features, effectively addressing these challenges while remaining computationally efficient. Experimental results from both subjective and objective evaluations demonstrate that DiTSE achieves state-of-the-art audio quality that, for the first time, matches real studio-quality audio from the DAPS dataset. Furthermore, DiTSE significantly improves the preservation of speaker identity and content fidelity, reducing hallucinations across datasets compared to state-of-the-art enhancers. Audio samples are available at: http://hguimaraes.me/DiTSE