mlsanityData to 7 Oct 2026

Are discrete units necessary for Spoken Language Modeling?

arXiv 2203.05936 ↗First posted 11 Mar 2022cs.CL cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Recent work in spoken language modeling shows the possibility of learning a language unsupervisedly from raw audio without any text labels. The approach relies first on transforming the audio into a sequence of discrete units (or pseudo-text) and then training a language model directly on such pseudo-text. Is such a discrete bottleneck necessary, potentially introducing irreversible errors in the encoding of the speech signal, or could we learn a language model without discrete units at all? In this work, we study the role of discrete versus continuous representations in spoken language modeling. We show that discretization is indeed essential for good results in spoken language modeling. We show that discretization removes linguistically irrelevant information from the continuous features, helping to improve language modeling performances. On the basis of this study, we train a language model on the discrete units of the HuBERT features, reaching new state-of-the-art results in the lexical, syntactic and semantic metrics of the Zero Resource Speech Challenge 2021 (Track 1 - Speech Only).

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

18 of 24 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  12. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
6 more
  1. 2020Unsupervised pretraining transfers well across languagescited by 100
  2. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  3. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  4. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  5. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  6. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 9; 2024: 2; 2025: 7.

24 papers.

Sort:

  1. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  5. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  6. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
  7. 2023RepCodec: A Speech Representation Codec for Speech Tokenizationcited by 27
  8. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  9. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  10. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  11. 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speechcited by 15
  12. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
12 more
  1. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  2. 2022Evaluating context-invariance in unsupervised speech representationscited by 11
  3. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
  4. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  5. 2025Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Modelscited by 4
  6. 2025Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0cited by 4
  7. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  8. 2022Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingscited by 2
  9. 2025Towards few-shot isolated word reading assessmentcited by 1
  10. 2025An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-trainingcited by 1
  11. 2023MASR: Multi-label Aware Speech Representationcited by 0
  12. 2024Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representationscited by 0