mlsanityData to 14 Oct 2026

SoundStorm: Efficient Parallel Audio Generation

arXiv 2305.09636 ↗First posted 16 May 2023cs.SD cs.LG eess.AScited by 114

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present SoundStorm, a model for efficient, non-autoregressive audio generation. SoundStorm receives as input the semantic tokens of AudioLM, and relies on bidirectional attention and confidence-based parallel decoding to generate the tokens of a neural audio codec. Compared to the autoregressive generation approach of AudioLM, our model produces audio of the same quality and with higher consistency in voice and acoustic conditions, while being two orders of magnitude faster. SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4. We demonstrate the ability of our model to scale audio generation to longer sequences by synthesizing high-quality, natural dialogue segments, given a transcript annotated with speaker turns and a short prompt with the speakers' voices.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Builds on 29 of its 33 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

29 of 33 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2020Reformer: The Efficient Transformercited by 1,668
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  12. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
17 more
  1. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  2. 2022High Fidelity Neural Audio Compressioncited by 687
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  5. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  6. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2017Non-Autoregressive Neural Machine Translationcited by 463
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  11. 2022AudioGen: Textually Guided Audio Generationcited by 295
  12. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  13. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  14. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  15. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  16. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  17. 2023SingSong: Generating musical accompaniments from singingcited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 49; 2025: 29; 2026: 27.

114 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  4. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  5. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  6. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  7. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  8. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  9. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  10. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  11. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  12. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
38 more
  1. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 78
  2. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  3. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  4. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  5. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  6. 2024BAMM: Bidirectional Autoregressive Motion Modelcited by 55
  7. 2023Efficient Neural Music Generationcited by 50
  8. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  9. 2024Foundation Models for Music: A Surveycited by 37
  10. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  11. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  12. 2023StemGen: A music generation model that listenscited by 32
  13. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  14. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  15. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  16. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  17. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  18. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  19. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 20
  20. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  21. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  22. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  23. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  24. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  25. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  26. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 15
  27. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  28. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  29. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  30. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  31. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  32. 2025Scaling Self-Supervised Representation Learning for Symbolic Piano Performancecited by 9
  33. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  34. 2024Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representationscited by 7
  35. 2024Takin: A Cohort of Superior Quality Zero-shot Speech Generation Modelscited by 7
  36. 2024Efficient Generative Modeling with Residual Vector Quantization-Based Tokenscited by 7
  37. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  38. 2026MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Trainingcited by 7

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation ↗
  2. 2026Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts
  3. 2025Continuous Audio Language Models
  4. 2026UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
  5. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
  6. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
  7. 2024AudioComposer: Towards Fine-grained Audio Generation with Natural Language Descriptions
  8. 2026Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text