mlsanityData to 7 Oct 2026

AudioLM: a Language Modeling Approach to Audio Generation

arXiv 2209.03143 ↗First posted 7 Sep 2022cs.SD cs.LG eess.AScited by 611

Not on the map. Why?

Cited by 64 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.

Key facts. Cited by 611 papers in the map (data through 7 Oct 2026). Builds on 57 of its 68 references that are in the map. First posted on arXiv 7 Sep 2022.

Builds on

57 of 68 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
38 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  11. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  12. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  13. 2020Jukebox: A Generative Model for Musiccited by 535
  14. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  15. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  16. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  17. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  18. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  19. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  20. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  21. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  22. 2019Deep Learning for Symbolic Mathematicscited by 266
  23. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  24. 2018Efficient Neural Audio Synthesiscited by 218
  25. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 215
  26. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  27. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  28. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  29. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  30. 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 90
  31. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 76
  32. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  33. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  34. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  35. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 35
  36. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  37. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  38. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 116; 2024: 188; 2025: 172; 2026: 120.

611 papers.

Sort:

By area:

  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2023ModelScope Text-to-Video Technical Reportcited by 651
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  4. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  5. 2023MusicLM: Generating Music From Textcited by 442
  6. 2023Learning Interactive Real-World Simulatorscited by 440
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  8. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  9. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  10. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  11. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
38 more
  1. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  2. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  3. 2022AudioGen: Textually Guided Audio Generationcited by 292
  4. 2023Towards Generalist Biomedical AIcited by 286
  5. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  6. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  7. 2023Self-Consuming Generative Models Go MADcited by 239
  8. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  9. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  10. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  11. 2025Kimi-Audio Technical Reportcited by 210
  12. 2022Continuous diffusion for categorical datacited by 202
  13. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  14. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  15. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  16. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  17. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  18. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  19. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 163
  20. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  21. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  22. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  23. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  24. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  25. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  26. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  27. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  28. 2024Large Language Models for Time Series: A Surveycited by 113
  29. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  30. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  31. 2025Step-Audio 2 Technical Reportcited by 101
  32. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  33. 2023Textually Pretrained Speech Language Modelscited by 99
  34. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  35. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  36. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  37. 2024Long-form music generation with latent diffusioncited by 88
  38. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86

50 most cited of 611 All 611 (JSON)