mlsanityData to 14 Oct 2026

DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models

arXiv 2410.24177 ↗First posted 31 Oct 2024eess.AS cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Spoken language models (SLMs) have gained increasing attention with advancements in text-based, decoder-only language models. SLMs process text and speech, enabling simultaneous speech understanding and generation. This paper presents Double-Codebook Speaker-invariant Clustering (DC-Spin), which aims to improve speech tokenization by bridging audio signals and SLM tokens. DC-Spin extracts speaker-invariant tokens rich in phonetic information and resilient to input variations, enhancing zero-shot SLM tasks and speech resynthesis. We propose a chunk-wise approach to enable streamable DC-Spin without retraining and degradation. Comparisons of tokenization methods (self-supervised and neural audio codecs), model scalability, and downstream task proxies show that tokens easily modeled by an n-gram LM or aligned with phonemes offer strong performance, providing insights for designing speech tokenizers for SLMs.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 65 of its 74 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

65 of 74 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
38 more
  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  6. 2022High Fidelity Neural Audio Compressioncited by 687
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  12. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  13. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  14. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  15. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  16. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  17. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  18. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  19. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  20. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  21. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  22. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  23. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  24. 2021Direct speech-to-speech translation with discrete unitscited by 149
  25. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  26. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  27. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 122
  28. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  29. 2023Textually Pretrained Speech Language Modelscited by 101
  30. 2023Joint Audio and Speech Understandingcited by 100
  31. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  32. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  33. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  34. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  35. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  36. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  37. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  38. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  3. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  4. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  5. 2025ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Modelscited by 4
  6. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  7. 2026USAD 2.0: Scaling Representation Distillation for Universal Audio Understandingcited by 1
  8. 2026Content is What Remains: Invariant Speech Tokenization from Parallel Utterancescited by 0
  9. 2026Estimating Surprisal from the Speech Signalcited by 0