mlsanityData to 7 Oct 2026

High Fidelity Neural Audio Compression

arXiv 2210.13438 ↗First posted 24 Oct 2022eess.AS cs.AI cs.SDcited by 673

Not on the map. Why?

Cited by 107 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

Key facts. Cited by 673 papers in the map (data through 7 Oct 2026). Builds on 50 of its 76 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

50 of 76 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  10. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  12. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
38 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  2. 2016End-to-end Optimized Image Compressioncited by 673
  3. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  4. 2020Jukebox: A Generative Model for Musiccited by 535
  5. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  6. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  7. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  8. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  9. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2018Efficient Neural Audio Synthesiscited by 218
  12. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  13. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  14. 2021Direct speech-to-speech translation with discrete unitscited by 148
  15. 2022Generative Spoken Dialogue Language Modelingcited by 125
  16. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  17. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  18. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
  19. 2019Music Source Separation in the Waveform Domaincited by 98
  20. 2018Learned Video Compressioncited by 89
  21. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 76
  22. 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 72
  23. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  24. 2022ICASSP 2022 Deep Noise Suppression Challengecited by 57
  25. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 47
  26. 2020Voice Separation with an Unknown Number of Multiple Speakerscited by 46
  27. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  28. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  29. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42
  30. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 35
  31. 2017Wavenet based low rate speech codingcited by 32
  32. 2020Real-time Speech Frequency Bandwidth Extensioncited by 22
  33. 2021Generative Speech Coding with Predictive Variance Regularizationcited by 18
  34. 2019A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNetcited by 16
  35. 2021GAN Vocoder: Multi-Resolution Discriminator Is All You Needcited by 16
  36. 2020Single channel voice separation for unknown number of speakers under reverberant and noisy settingscited by 10
  37. 2019Improving Opus Low Bit Rate Quality with Neural Speech Synthesiscited by 9
  38. 2022End-to-End Neural Speech Coding for Real-Time Communicationscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 79; 2024: 215; 2025: 217; 2026: 162.

673 papers.

Sort:

By area:

  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  4. 2023Simple and Controllable Music Generationcited by 480
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  8. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  9. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  11. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  12. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
38 more
  1. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  2. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  4. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  5. 2024Stable Audio Opencited by 192
  6. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  7. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  8. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  9. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  10. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  11. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  12. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  13. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  14. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  15. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  16. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  17. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  18. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  19. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 115
  20. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  21. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  22. 2026Qwen3-TTS Technical Reportcited by 106
  23. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  24. 2025Step-Audio 2 Technical Reportcited by 101
  25. 2023Textually Pretrained Speech Language Modelscited by 99
  26. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  27. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  28. 2024Long-form music generation with latent diffusioncited by 88
  29. 2024ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scalecited by 88
  30. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  31. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  32. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  33. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  34. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  35. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  36. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 77
  37. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  38. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73

50 most cited of 673 All 673 (JSON)