mlsanityData to 14 Oct 2026

High-Fidelity Audio Compression with Improved RVQGAN

arXiv 2306.06546 ↗First posted 11 Jun 2023cs.SD cs.LG eess.AScited by 431

Not on the map. Why?

Cited by 84 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.

Key facts. Cited by 431 papers in the map (data through 14 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

39 of 47 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
27 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  2. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  3. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  4. 2022High Fidelity Neural Audio Compressioncited by 687
  5. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  8. 2020Jukebox: A Generative Model for Musiccited by 542
  9. 2023MusicLM: Generating Music From Textcited by 448
  10. 2018SDR - half-baked or well done?cited by 423
  11. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  12. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  13. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  14. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  15. 2017Geometric GANcited by 233
  16. 2020DDSP: Differentiable Digital Signal Processingcited by 178
  17. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  18. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  19. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 100
  20. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 79
  21. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 74
  22. 2020Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speechcited by 63
  23. 2022ICASSP 2022 Deep Noise Suppression Challengecited by 60
  24. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 49
  25. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42
  26. 2021Chunked Autoregressive GAN for Conditional Waveform Synthesiscited by 38
  27. 2020Upsampling artifacts in neural audio synthesiscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 112; 2025: 161; 2026: 150.

431 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  3. 2024Stable Audio Opencited by 197
  4. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  5. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  6. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  7. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  8. 2026Qwen3-TTS Technical Reportcited by 118
  9. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  10. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  11. 2024Long-form music generation with latent diffusioncited by 89
  12. 2024Recent Advances in Speech Language Models: A Surveycited by 87
38 more
  1. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 78
  2. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  3. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  4. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  5. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  6. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  7. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  8. 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
  9. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  10. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  11. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 48
  12. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 46
  13. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  14. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  15. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  16. 2024SNAC: Multi-Scale Neural Audio Codeccited by 43
  17. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  18. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
  19. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
  20. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 40
  21. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
  22. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 38
  23. 2024Foundation Models for Music: A Surveycited by 37
  24. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  25. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  26. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  27. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  28. 2024Music2Latent: Consistency Autoencoders for Latent Audio Compressioncited by 33
  29. 2023StemGen: A music generation model that listenscited by 32
  30. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 32
  31. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  32. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  33. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 31
  34. 2025SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generationcited by 29
  35. 2025VibeVoice Technical Reportcited by 29
  36. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  37. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
  38. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26

50 most cited of 431 All 431 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025QINCODEC: Neural Audio Compression with Implicit Neural Codebooks
  2. 2021Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization
  3. 2025Continuous Audio Language Models
  4. 2024A Survey of Deep Learning Audio Generation Methods
  5. 2021VQCPC-GAN: Variable-Length Adversarial Audio Synthesis Using Vector-Quantized Contrastive Predictive Coding
  6. 2026AudioTQ: A Data-Oblivious 6-Bit CPU Audio Codec via Randomized Hadamard Rotation and Lloyd-Max Quantization
  7. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  8. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation