mlsanityData to 7 Oct 2026

Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference

arXiv 2409.12117 ↗First posted 18 Sep 2024eess.AS cs.CL cs.SDcited by 11

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modeling techniques to audio data. However, audio codecs often operate at high frame rates, resulting in slow training and inference, especially for autoregressive models. To address this challenge, we present the Low Frame-rate Speech Codec (LFSC): a neural audio codec that leverages finite scalar quantization and adversarial training with large speech language models to achieve high-quality audio compression with a 1.89 kbps bitrate and 21.5 frames per second. We demonstrate that our novel codec can make the inference of LLM-based text-to-speech models around three times faster while improving intelligibility and producing quality comparable to previous models.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

32 of 40 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  12. 2023Simple and Controllable Music Generationcited by 480
20 more
  1. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  2. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  3. 2018SDR - half-baked or well done?cited by 420
  4. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  5. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  6. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  7. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  8. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  9. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  10. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 56
  11. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 47
  12. 2017End-to-End Optimized Speech Coding with Deep Neural Networkscited by 21
  13. 2023Fewer-token Neural Speech Codec with Time-invariant Codescited by 21
  14. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 21
  15. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  16. 2024Towards High-Quality and Efficient Speech Bandwidth Extension with Parallel Amplitude and Phase Predictioncited by 14
  17. 2024APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decodingcited by 12
  18. 2022Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translationcited by 9
  19. 2024Gull: A Generative Multifunctional Audio Codeccited by 5
  20. 2024An Intra-BRNN and GB-RVQ Based END-TO-END Neural Audio Codeccited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  2. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  3. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 26
  4. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 21
  5. 2025NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inferencecited by 14
  6. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
  7. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  8. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  9. 2026CrossAccent-TTS: Cross-Lingual Accent-Intensity Controllable Text-to-Speech via Disentangled Speaker and Accent Representationscited by 1
  10. 2026S-PRESSO: Ultra Low Bitrate Sound Effect Compression With Diffusion Autoencoders And Offline Quantizationcited by 0
  11. 2026Q-SPT: Learnable Query-Based Compression for Low-Frame-Rate Speech Tokenizationcited by 0