# LL-SDR: Low-Latency Speech enhancement through Discrete Representations

arXiv: https://arxiv.org/abs/2603.20242 · HTML: https://mlsanity.com/paper/2603.20242/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 27 of 27)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed](https://mlsanity.com/paper/1909.01174/) (2019)
- [SELM: Speech Enhancement Using Discrete Tokens and Language Models](https://mlsanity.com/paper/2312.09747/) (2023)
- [A scalable noisy speech dataset and online subjective test framework](https://mlsanity.com/paper/1909.08050/) (2019)
- [VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration](https://mlsanity.com/paper/2204.05841/) (2022)
- [FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancement](https://mlsanity.com/paper/2206.07293/) (2022)
- [Revisiting Over-Smoothness in Text to Speech](https://mlsanity.com/paper/2202.13066/) (2022)
- [InfoNCE: Identifying the Gap Between Theory and Practice](https://mlsanity.com/paper/2407.00143/) (2024)
- [MaskSR: Masked Language Model for Full-band Speech Restoration](https://mlsanity.com/paper/2406.02092/) (2024)
- [GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling](https://mlsanity.com/paper/2502.02942/) (2025)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement](https://mlsanity.com/paper/2503.00493/) (2025)
- [An EM Approach to Non-autoregressive Conditional Sequence Generation](https://mlsanity.com/paper/2006.16378/) (2020)
- [Anytime Sampling for Autoregressive Models via Ordered Autoencoding](https://mlsanity.com/paper/2102.11495/) (2021)
- [SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis](https://mlsanity.com/paper/2409.00933/) (2024)
- [Speech Enhancement Using Continuous Embeddings of Neural Audio Codec](https://mlsanity.com/paper/2502.16240/) (2025)
- [An Investigation into the Effectiveness of Enhancement in ASR Training and Test for CHiME-5 Dinner Party Transcription](https://mlsanity.com/paper/1909.12208/) (2019)
- [FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation](https://mlsanity.com/paper/2509.16195/) (2025)
- [An individualized super Gaussian single microphone Speech Enhancement for hearing aid users with smartphone as an assistive device](https://mlsanity.com/paper/1812.03916/) (2018)

## Abstract (arXiv)

Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches. Despite its strong enhancement performance, LL-SDR remains lightweight and efficient, requiring only 40G MACs for a single forward pass on a 10-second 16 kHz speech segment and achieving low-latency inference with an RTF of 0.01 on GPU and 0.24 on CPU. Demos and source code are available at our project websites.