mlsanityData to 14 Oct 2026

Generative Spoken Language Modeling from Raw Audio

arXiv 2102.01192 ↗First posted 1 Feb 2021cs.CLcited by 340

Not on the map. Why?

Abstract (arXiv)

We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo-text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder-dependent way, and that some combinations approach text-based systems.

Key facts. Cited by 340 papers in the map (data through 14 Oct 2026). Builds on 55 of its 79 references that are in the map. First posted on arXiv 1 Feb 2021.

Builds on

55 of 79 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
38 more
  1. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  2. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  5. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  6. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  7. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  8. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  9. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  10. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  11. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  12. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 130
  13. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  14. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  15. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 93
  16. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  17. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  18. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  19. 2016Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learnercited by 68
  20. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  21. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  22. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 59
  23. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  24. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
  25. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  26. 2020The Zero Resource Speech Challenge 2020: Discovering discrete subword and word unitscited by 39
  27. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  28. 2020Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstructioncited by 38
  29. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  30. 2017Learning Latent Representations for Speech Generation and Transformationcited by 27
  31. 2019VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019cited by 25
  32. 2020Improved Speech Representations with Multi-Target Autoregressive Predictive Codingcited by 23
  33. 2020Unsupervised Cross-Domain Singing Voice Conversioncited by 23
  34. 2020Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challengecited by 21
  35. 2020Self-Supervised Representations Improve End-to-End Speech Translationcited by 21
  36. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  37. 2020A Convolutional Deep Markov Model for Unsupervised Speech Representation Learningcited by 18
  38. 2019TTS Skins: Speaker Conversion via ASRcited by 14

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 44; 2023: 78; 2024: 84; 2025: 75; 2026: 41.

340 papers.

Sort:

By area:

  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  2. 2022High Fidelity Neural Audio Compressioncited by 687
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  4. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  6. 2023Simple and Controllable Music Generationcited by 488
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  8. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  9. 2022AudioGen: Textually Guided Audio Generationcited by 295
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  11. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  12. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
38 more
  1. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  2. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  3. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  4. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  5. 2021Direct speech-to-speech translation with discrete unitscited by 149
  6. 2022Generative Spoken Dialogue Language Modelingcited by 126
  7. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  8. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  9. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  10. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  11. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  12. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  13. 2023Textually Pretrained Speech Language Modelscited by 101
  14. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  15. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 99
  16. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  17. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  18. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  19. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  20. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  21. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  22. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  23. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  24. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  25. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 72
  26. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 67
  27. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  28. 2024Language Model Can Listen While Speakingcited by 63
  29. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  30. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  31. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  32. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  33. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  34. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  35. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  36. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  37. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 44
  38. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43

50 most cited of 340 All 340 (JSON)