mlsanityData to 7 Oct 2026

Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling

arXiv 2301.00591 ↗First posted 2 Jan 2023cs.CL cs.SD eess.AScited by 47

Not on the map. Why?

Abstract (arXiv)

This work profoundly analyzes discrete self-supervised speech representations (units) through the eyes of Generative Spoken Language Modeling (GSLM). Following the findings of such an analysis, we propose practical improvements to the discrete unit for the GSLM. First, we start comprehending these units by analyzing them in three axes: interpretation, visualization, and resynthesis. Our analysis finds a high correlation between the speech units to phonemes and phoneme families, while their correlation with speaker or gender is weaker. Additionally, we found redundancies in the extracted units and claim that one reason may be the units' context. Following this analysis, we propose a new, unsupervised metric to measure unit redundancies. Finally, we use this metric to develop new methods that improve the robustness of units' clustering and show significant improvement considering zero-resource speech metrics such as ABX. Code and analysis tools are available under the following link: https://github.com/slp-rl/SLM-Discrete-Representations

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 2 Jan 2023.

Builds on

15 of 24 references.

References · 15

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  8. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2022Generative Spoken Dialogue Language Modelingcited by 125
  11. 2020Unsupervised pretraining transfers well across languagescited by 100
  12. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
3 more
  1. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 31
  2. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  3. 2022Self-supervised Speaker Diarizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 10; 2025: 17; 2026: 9.

47 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2023Textually Pretrained Speech Language Modelscited by 99
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  4. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  5. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  6. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  7. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  8. 2023Self-Supervised Models of Speech Infer Universal Articulatory Kinematicscited by 16
  9. 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speechcited by 15
  10. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  11. 2023SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERTcited by 13
  12. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13
35 more
  1. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  2. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
  3. 2024UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalizationcited by 8
  4. 2024How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representationscited by 8
  5. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7
  6. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 7
  7. 2025The Role of Prosody in Spoken Question Answeringcited by 6
  8. 2025PAST: Phonetic-Acoustic Speech Tokenizercited by 6
  9. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  10. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  11. 2024LVCHAT: Facilitating Long Video Comprehensioncited by 4
  12. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  13. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  14. 2023In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesiscited by 3
  15. 2023Non Intrusive Intelligibility Predictor for Hearing Impaired Individuals using Self Supervised Speech Representationscited by 3
  16. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  17. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  18. 2026Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTScited by 3
  19. 2025Entropy-based Coarse and Compressed Semantic Speech Representation Learningcited by 2
  20. 2025Analysing the Language of Neural Audio Codecscited by 2
  21. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  22. 2026Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Modelscited by 2
  23. 2023Towards Matching Phones and Speech Representationscited by 1
  24. 2024Discrete Unit based Masking for Improving Disentanglement in Voice Conversioncited by 1
  25. 2025Unsupervised Speech Segmentation: A General Approach Using Speech Language Modelscited by 1
  26. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  27. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modelingcited by 1
  28. 2026Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditionscited by 1
  29. 2024Enhancing Code-switched Text-to-Speech Synthesis Capability in Large Language Models with only Monolingual Corporacited by 0
  30. 2025Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Taskscited by 0
  31. 2025GmSLM : Generative Marmoset Spoken Language Modelingcited by 0
  32. 2026On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluationcited by 0
  33. 2026Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgettingcited by 0
  34. 2026UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perceptioncited by 0
  35. 2026Phone Segmentation and Recognition through Phonological Activation Mappingcited by 0