mlsanityData to 7 Oct 2026

SoundStream: An End-to-End Neural Audio Codec

arXiv 2107.03312 ↗First posted 7 Jul 2021cs.SD cs.LG eess.AScited by 756

Not on the map. Why?

Cited by 92 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. By training with structured dropout applied to quantizer layers, a single model can operate across variable bitrates from 3kbps to 18kbps, with a negligible quality loss when compared with models trained at fixed bitrates. In addition, the model is amenable to a low latency implementation, which supports streamable inference and runs in real time on a smartphone CPU. In subjective evaluations using audio at 24kHz sampling rate, SoundStream at 3kbps outperforms Opus at 12kbps and approaches EVS at 9.6kbps. Moreover, we are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency, which we demonstrate through background noise suppression for speech.

Key facts. Cited by 756 papers in the map (data through 7 Oct 2026). Builds on 34 of its 60 references that are in the map. First posted on arXiv 7 Jul 2021.

Builds on

34 of 60 references.

References · 34

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2017The Perception-Distortion Tradeoffcited by 582
  10. 2020Jukebox: A Generative Model for Musiccited by 535
  11. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  12. 2020High-Fidelity Generative Image Compressioncited by 331
22 more
  1. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  2. 2018Efficient Neural Audio Synthesiscited by 218
  3. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  4. 2020DDSP: Differentiable Digital Signal Processingcited by 177
  5. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 165
  6. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  7. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
  8. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 76
  9. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  10. 2017A Wavenet for Speech Denoisingcited by 47
  11. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 47
  12. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42
  13. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 35
  14. 2017Wavenet based low rate speech codingcited by 32
  15. 2018Speech Denoising with Deep Feature Lossescited by 26
  16. 2020Real-time Speech Frequency Bandwidth Extensioncited by 22
  17. 2017End-to-End Optimized Speech Coding with Deep Neural Networkscited by 21
  18. 2017Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognitioncited by 19
  19. 2021Generative Speech Coding with Predictive Variance Regularizationcited by 18
  20. 2019Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Codingcited by 15
  21. 2020Audio Codec Enhancement with Generative Adversarial Networkscited by 11
  22. 2021Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoderscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 30; 2023: 99; 2024: 205; 2025: 240; 2026: 178.

756 papers.

Sort:

By area:

  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2022High Fidelity Neural Audio Compressioncited by 673
  3. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  6. 2023Simple and Controllable Music Generationcited by 480
  7. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  8. 2023MusicLM: Generating Music From Textcited by 442
  9. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  10. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  11. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  12. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
38 more
  1. 2023Compact 3D Gaussian Representation for Radiance Fieldcited by 365
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  3. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  4. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  5. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  6. 2022AudioGen: Textually Guided Audio Generationcited by 292
  7. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  9. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  10. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  11. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  12. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  13. 2023Recommender Systems with Generative Retrievalcited by 191
  14. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  15. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 166
  16. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  17. 2024Behavior Generation with Latent Actionscited by 159
  18. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  19. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  20. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  21. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  22. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 140
  23. 2021Taming Visually Guided Sound Generationcited by 136
  24. 2024Recent Advances in 3D Gaussian Splattingcited by 130
  25. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  26. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  27. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  28. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  29. 20234M: Massively Multimodal Masked Modelingcited by 118
  30. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  31. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  32. 2023Learning to Act without Actionscited by 116
  33. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  34. 2024Large Language Models for Time Series: A Surveycited by 113
  35. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  36. 2026Qwen3-TTS Technical Reportcited by 106
  37. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  38. 20243D Gaussian as a New Era: A Surveycited by 102

50 most cited of 756 All 756 (JSON)