mlsanityData to 19 Oct 2026

PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning

arXiv 2511.22687 ↗First posted 27 Nov 2025cs.SD eess.AScited by 1

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Neural speech codecs have achieved strong performance in low-bitrate compression, but residual vector quantization (RVQ) often suffers from unstable training and ineffective decomposition, limiting reconstruction quality and efficiency. We propose PURE Codec (Progressive Unfolding of Residual Entropy), a novel framework that guides multi-stage quantization using a pre-trained speech enhancement model. The first quantization stage reconstructs low-entropy, denoised speech embeddings, while subsequent stages encode residual high-entropy components. This design improves training stability significantly. Experiments demonstrate that PURE consistently outperforms conventional RVQ-based codecs in reconstruction and downstream speech language model-based text-to-speech, particularly under noisy training conditions.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 48 of its 61 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

48 of 61 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  2. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Coding29 Nov
  3. 2024TS3-Codec: Transformer-Based Simple Streaming Single Codec27 Nov
  4. 2024SNAC: Multi-Scale Neural Audio Codec18 Oct
  5. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech24 Sep
  6. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference18 Sep
  7. 2024BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec9 Sep
  8. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling29 Aug
  9. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens7 Jul
  10. 2024NAST: Noise Aware Speech Tokenization for Speech Language Models16 Jun
  11. 2024HILCodec: High-Fidelity and Lightweight Neural Audio Codec8 May
  12. 2024ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers30 Apr
8 more
  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  2. 2024Towards audio language modeling -- an overview20 Feb
  3. 2023Generative De-Quantization for Neural Speech Codec via Latent Diffusion14 Nov
  4. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec14 Sep
  5. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models31 Aug
  6. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec26 May
  7. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec4 May
  8. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 28

  1. 2017Neural Discrete Representation Learningcited by 4,843
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,596
  4. 2022High Fidelity Neural Audio Compressioncited by 917
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 864
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 711
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
  8. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  9. 2023Simple and Controllable Music Generationcited by 540
  10. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 292
  11. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 246
  12. 2022TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separationcited by 121
16 more
  1. 2024Recent Advances in Speech Language Models: A Surveycited by 106
  2. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 101
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 65
  4. 2024Codec-SUPERB: An In-Depth Analysis of Sound Codec Modelscited by 52
  5. 2024URGENT Challenge: Universality, Robustness, and Generalizability For Speech Enhancementcited by 44
  6. 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 44
  7. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 42
  8. 2024UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learnercited by 41
  9. 2024APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decodingcited by 40
  10. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 21
  11. 2022Disentangled Feature Learning for Real-Time Neural Speech Codingcited by 17
  12. 2021HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Codingcited by 14
  13. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 14
  14. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 11
  15. 2020Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Losscited by 10
  16. 2024Scale This, Not That: Investigating Key Dataset Attributes for Efficient Speech Enhancement Scalingcited by 2

Built on by

Cited by year (2026 to 19 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TokenMapper: A Step Toward Interoperable Speech Token Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Entropy-Guided GRVQ for Ultra-Low Bitrate Neural Speech Codec
  2. 2026An Ultra-Low-Bitrate Neural Speech Codec with Plain-to-Pseudo Synergistic Vector Quantization
  3. 2024Language-Codec: Bridging Discrete Codec Representations and Speech Language Models
  4. 2024ERVQ: Enhanced Residual Vector Quantization with Intra-and-Inter-Codebook Optimization for Neural Audio Codecs
  5. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  6. 2025MBCodec:Thorough disentangle for high-fidelity audio compression
  7. 2026Benchmarking Neural Speech Compression from a Rate-Distortion Perspective
  8. 2025Decoder-Side Semantic Conditioning for Low-Bitrate Neural Speech Compression