mlsanityData to 7 Oct 2026

U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation

arXiv 2510.16718 ↗First posted 19 Oct 2025cs.SD cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We propose U-Codec, an Ultra low frame-rate neural speech Codec that achieves high-fidelity reconstruction and fast speech generation at an extremely low frame-rate of 5Hz (5 frames per second). Extreme compression at 5Hz typically leads to severe intelligibility and spectral detail loss, we introduce a Transformer-based inter-frame long-term dependency module and systematically explore residual vector quantization (RVQ) depth and codebook size to identify optimal configurations. Moreover, we apply U-Codec into a large language model (LLM)-based auto-regressive TTS model, which leverages global and local hierarchical architecture to effectively capture dependencies across multi-layer tokens. We extend LLM-based TTS from 3-layer RVQ at 50Hz to 32-layer RVQ at 5Hz. Experimental results demonstrate that U-Codec improves LLM-based TTS inference speed by around 3 × over high-frame-rate codecs while maintaining similarity and naturalness. These results validate the feasibility of using highly compressed 5Hz discrete tokens for fast and high-fidelity speech synthesis.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 46 of its 50 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

46 of 50 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  3. 2016Least Squares Generative Adversarial Networkscited by 1,000
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  6. 2022High Fidelity Neural Audio Compressioncited by 673
  7. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  11. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  12. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
34 more
  1. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  2. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  3. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  4. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  5. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  6. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  7. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  8. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  9. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  10. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  11. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  12. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  13. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  14. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  15. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  16. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
  17. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
  18. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  19. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  20. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 48
  21. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
  22. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
  23. 2018Sample Efficient Adaptive Text-to-Speechcited by 45
  24. 2024BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codeccited by 45
  25. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  26. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speechcited by 37
  27. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  28. 2025Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglementcited by 34
  29. 2020Upsampling artifacts in neural audio synthesiscited by 28
  30. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  31. 2024UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learnercited by 22
  32. 2025DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generationcited by 20
  33. 2025ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modelingcited by 12
  34. 2024HILCodec: High-Fidelity and Lightweight Neural Audio Codeccited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Codingcited by 0