# SELM: Speech Enhancement Using Discrete Tokens and Language Models

arXiv: https://arxiv.org/abs/2312.09747 · HTML: https://mlsanity.com/paper/2312.09747/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2023.

## Built on by (most cited first, 30 of 36)

- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [MaskSR: Masked Language Model for Full-band Speech Restoration](https://mlsanity.com/paper/2406.02092/) (2024)
- [How Should We Extract Discrete Audio Tokens from Self-Supervised Models?](https://mlsanity.com/paper/2406.10735/) (2024)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching](https://mlsanity.com/paper/2505.19476/) (2025)
- [TSELM: Target Speaker Extraction using Discrete Tokens and Language Models](https://mlsanity.com/paper/2409.07841/) (2024)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [Speech Enhancement Using Continuous Embeddings of Neural Audio Codec](https://mlsanity.com/paper/2502.16240/) (2025)
- [DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformers](https://mlsanity.com/paper/2504.09381/) (2025)
- [Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility](https://mlsanity.com/paper/2409.09357/) (2024)
- [DM-Codec: Distilling Multimodal Representations for Speech Tokenization](https://mlsanity.com/paper/2410.15017/) (2024)
- [LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Models](https://mlsanity.com/paper/2504.07402/) (2025)
- [FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learning](https://mlsanity.com/paper/2601.16483/) (2026)
- [UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement](https://mlsanity.com/paper/2510.20441/) (2025)
- [Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization](https://mlsanity.com/paper/2601.23174/) (2026)
- [Aligning Generative Speech Enhancement with Perceptual Feedback](https://mlsanity.com/paper/2507.09929/) (2025)
- [SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement](https://mlsanity.com/paper/2509.24708/) (2025)
- [From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodology](https://mlsanity.com/paper/2412.17778/) (2024)
- [MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement](https://mlsanity.com/paper/2507.00966/) (2025)
- [High-Fidelity Speech Enhancement via Discrete Audio Tokens](https://mlsanity.com/paper/2510.02187/) (2025)
- [Autoregressive Speech Enhancement via Acoustic Tokens](https://mlsanity.com/paper/2507.12825/) (2025)
- [GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model](https://mlsanity.com/paper/2512.20978/) (2025)
- [Post-Training Speech Enhancement Language Models with Perceptual Rewards](https://mlsanity.com/paper/2606.21458/) (2026)
- [aTENNuate: Optimized Real-time Speech Enhancement with Deep SSMs on Raw Audio](https://mlsanity.com/paper/2409.03377/) (2024)
- [Are Deep Speech Denoising Models Robust to Adversarial Noise?](https://mlsanity.com/paper/2503.11627/) (2025)
- [SLM-SS: Speech Language Model for Generative Speech Separation](https://mlsanity.com/paper/2601.19533/) (2026)
- [Frontend Token Enhancement for Token-Based Speech Recognition](https://mlsanity.com/paper/2602.04217/) (2026)
- [Absorbing Discrete Diffusion for Speech Enhancement](https://mlsanity.com/paper/2602.22417/) (2026)

## Abstract (arXiv)

Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception. Our demos are available https://honee-w.github.io/SELM/.