mlsanityData to 14 Oct 2026

Estimating the Completeness of Discrete Speech Units

arXiv 2409.06109 ↗First posted 9 Sep 2024eess.AS cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Representing speech with discrete units has been widely used in speech codec and speech generation. However, there are several unverified claims about self-supervised discrete units, such as disentangling phonetic and speaker information with k-means, or assuming information loss after k-means. In this work, we take an information-theoretic perspective to answer how much information is present (information completeness) and how much information is accessible (information accessibility), before and after residual vector quantization. We show a lower bound for information completeness and estimate completeness on discretized HuBERT representations after residual vector quantization. We find that speaker information is sufficiently present in HuBERT discrete units, and that phonetic information is sufficiently present in the residual, showing that vector quantization does not achieve disentanglement. Our results offer a comprehensive assessment on the choice of discrete units, and suggest that a lot more information in the residual should be mined rather than discarded.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

39 of 48 references.

References · 39

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  12. 2022High Fidelity Neural Audio Compressioncited by 687
27 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  4. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  5. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  6. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  7. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  8. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  9. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  10. 2021Direct speech-to-speech translation with discrete unitscited by 149
  11. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  12. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 91
  13. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  14. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  15. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  16. 2023Voice Conversion With Just Nearest Neighborscited by 68
  17. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  18. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  19. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  20. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  21. 2024Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Modelscited by 27
  22. 2023Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTScited by 24
  23. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 17
  24. 2019Adaptive Density Estimation for Generative Modelscited by 16
  25. 2023Quantifying the redundancy between prosody and textcited by 12
  26. 2024Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspectivecited by 5
  27. 2023Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 5.

9 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 9
  3. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  4. 2026Rethinking Discrete Speech Representation Tokens for Accent Generationcited by 2
  5. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  6. 2026DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Unitscited by 0
  7. 2026MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variablescited by 0
  8. 2026Content is What Remains: Invariant Speech Tokenization from Parallel Utterancescited by 0
  9. 2026A Comprehensive Study of Content Representations for Speech Synthesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speech
  2. 2022Autoregressive Co-Training for Learning Discrete Speech Representations
  3. 2021Discrete representations in neural models of spoken language
  4. 2026Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'a
  5. 2022Self-supervised speech unit discovery from articulatory and acoustic features using VQ-VAE
  6. 2024Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling
  7. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR
  8. 2021Learning De-identified Representations of Prosody from Raw Audio