mlsanityData to 14 Oct 2026

Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate

arXiv 2505.16845 ↗First posted 22 May 2025eess.AS cs.AI cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Most neural speech codecs achieve bitrate adjustment through intra-frame mechanisms, such as codebook dropout, at a Constant Frame Rate (CFR). However, speech segments inherently have time-varying information density (e.g., silent intervals versus voiced regions). This property makes CFR not optimal in terms of bitrate and token sequence length, hindering efficiency in real-time applications. In this work, we propose a Temporally Flexible Coding (TFC) technique, introducing variable frame rate (VFR) into neural speech codecs for the first time. TFC enables seamlessly tunable average frame rates and dynamically allocates frame rates based on temporal entropy. Experimental results show that a codec with TFC achieves optimal reconstruction quality with high flexibility, and maintains competitive performance even at lower frame rates. Our approach is promising for the integration with other efforts to develop low-frame-rate neural speech codecs for more efficient downstream tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 33 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

31 of 33 references.

References · 31

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2022High Fidelity Neural Audio Compressioncited by 687
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  8. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  9. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  11. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  12. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
19 more
  1. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  2. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  3. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 60
  4. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 49
  5. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  6. 2024SNAC: Multi-Scale Neural Audio Codeccited by 43
  7. 2024Codec-SUPERB: An In-Depth Analysis of Sound Codec Modelscited by 26
  8. 2024TS3-Codec: Transformer-Based Simple Streaming Single Codeccited by 25
  9. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  10. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generationcited by 23
  11. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  12. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  13. 2024Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaptationcited by 16
  14. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inferencecited by 11
  15. 2024Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspectivecited by 8
  16. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
  17. 2024Variable Bitrate Residual Vector Quantization for Audio Codingcited by 6
  18. 2024TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizerscited by 4
  19. 2024Speaking from Coarse to Fine: Improving Neural Codec Language Model via Multi-Scale Speech Coding and Generationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  2. 2026Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Codingcited by 0
  3. 2026TokenMapper: A Step Toward Interoperable Speech Token Translationcited by 0
  4. 2026LACE: Layer-Wise Compression for Dynamic Frame Rate Codecscited by 0