mlsanityData to 7 Oct 2026

SELM: Speech Enhancement Using Discrete Tokens and Language Models

arXiv 2312.09747 ↗First posted 15 Dec 2023eess.AS eess.SPcited by 36

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception. Our demos are available https://honee-w.github.io/SELM/.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Dec 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 15; 2026: 12.

36 papers.

Sort:

By area:

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  3. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  4. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  5. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  6. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  7. 2025FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matchingcited by 17
  8. 2024TSELM: Target Speaker Extraction using Discrete Tokens and Language Modelscited by 10
  9. 2024FINALLY: fast and universal speech enhancement with studio-like qualitycited by 7
  10. 2025Speech Enhancement Using Continuous Embeddings of Neural Audio Codeccited by 7
  11. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
  12. 2024Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibilitycited by 5
24 more
  1. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5
  2. 2025LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Modelscited by 5
  3. 2026FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learningcited by 5
  4. 2025UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancementcited by 4
  5. 2025SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancementcited by 3
  6. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 3
  7. 2024From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodologycited by 2
  8. 2025MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancementcited by 2
  9. 2025Aligning Generative Speech Enhancement with Perceptual Feedbackcited by 2
  10. 2025High-Fidelity Speech Enhancement via Discrete Audio Tokenscited by 2
  11. 2025Autoregressive Speech Enhancement via Acoustic Tokenscited by 1
  12. 2025GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Modelcited by 1
  13. 2026Post-Training Speech Enhancement Language Models with Perceptual Rewardscited by 1
  14. 2024aTENNuate: Optimized Real-time Speech Enhancement with Deep SSMs on Raw Audiocited by 0
  15. 2025Are Deep Speech Denoising Models Robust to Adversarial Noise?cited by 0
  16. 2026SLM-SS: Speech Language Model for Generative Speech Separationcited by 0
  17. 2026Frontend Token Enhancement for Token-Based Speech Recognitioncited by 0
  18. 2026Absorbing Discrete Diffusion for Speech Enhancementcited by 0
  19. 2026TokenSE: a Mamba-based discrete token speech enhancement framework for cochlear implantscited by 0
  20. 2026UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinationscited by 0
  21. 2026UR-BERT: Scaling Text Encoders for Massively Multilingual TTS Through Universal Romanization and Speech Token Predictioncited by 0
  22. 2026PhASE-Flow: Phonetic-Conditioned Acoustic Flow Matching in SSL Representation Domain for Speech Enhancementcited by 0
  23. 2026Test-time adaptation for speech enhancement with an autoregressive speech priorcited by 0
  24. 2026Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representationscited by 0