mlsanityData to 7 Oct 2026

TSELM: Target Speaker Extraction using Discrete Tokens and Language Models

arXiv 2409.07841 ↗First posted 12 Sep 2024cs.SD cs.LG eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

We propose TSELM, a novel target speaker extraction network that leverages discrete tokens and language models. TSELM utilizes multiple discretized layers from WavLM as input tokens and incorporates cross-attention mechanisms to integrate target speaker information. Language models are employed to capture the sequence dependencies, while a scalable HiFi-GAN is used to reconstruct the audio from the tokens. By applying a cross-entropy loss, TSELM models the probability distribution of output tokens, thus converting the complex regression problem of audio generation into a classification task. Experimental results show that TSELM achieves excellent results in speech quality and comparable results in speech intelligibility.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

30 of 36 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  10. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  11. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
18 more
  1. 2022AudioGen: Textually Guided Audio Generationcited by 292
  2. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  3. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  4. 2020Attention is All You Need in Speech Separationcited by 184
  5. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 165
  6. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  7. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
  8. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  9. 2023Voice Conversion With Just Nearest Neighborscited by 67
  10. 2023SELM: Speech Enhancement Using Discrete Tokens and Language Modelscited by 36
  11. 2020SpEx+: A Complete Time Domain Speaker Extraction Networkcited by 33
  12. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  13. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  14. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  15. 2019Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modelingcited by 21
  16. 2023TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognitioncited by 8
  17. 2024USEF-TSE: Universal Speaker Embedding Free Target Speaker Extractioncited by 5
  18. 2023Conditional Diffusion Model for Target Speaker Extractioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  3. 2025FlowTSE: Target Speaker Extraction with Flow Matchingcited by 6
  4. 2025LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Modelscited by 5
  5. 2025UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancementcited by 4
  6. 2025GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Modelcited by 1
  7. 2026StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Modelcited by 1
  8. 2026SLM-SS: Speech Language Model for Generative Speech Separationcited by 0
  9. 2026Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matchingcited by 0
  10. 2026Breaking the Quality--Intelligibility Trade-off in Streaming Target Speaker Extraction via Deep-Feature-Anchored Preference Optimizationcited by 0