mlsanityData to 14 Oct 2026

Scaling Transformers for Low-Bitrate High-Quality Speech Coding

arXiv 2411.19842 ↗First posted 29 Nov 2024eess.AS cs.AI cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

The tokenization of speech with neural audio codec models is a vital part of modern AI pipelines for the generation or understanding of speech, alone or in a multimodal context. Traditionally such tokenization models have concentrated on low parameter-count architectures using only components with strong inductive biases. In this work we show that by scaling a transformer architecture with large parameter count to this problem, and applying a flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to reach state-of-the-art speech quality at extremely low bit-rates of 400 or 700 bits-per-second. The trained models strongly out-perform existing baselines in both objective and subjective tests.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

47 of 55 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
35 more
  1. 2021Going deeper with Image Transformerscited by 764
  2. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  3. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  4. 2022High Fidelity Neural Audio Compressioncited by 687
  5. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  8. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  10. 2023Simple and Controllable Music Generationcited by 488
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  13. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  14. 2018SDR - half-baked or well done?cited by 423
  15. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  16. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  17. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  18. 2018Adversarial Audio Synthesiscited by 225
  19. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  20. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  21. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 127
  22. 2024Long-form music generation with latent diffusioncited by 89
  23. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  24. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  25. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  26. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesiscited by 37
  27. 2022Full-band General Audio Synthesis with Score-based Diffusioncited by 32
  28. 2023StemGen: A music generation model that listenscited by 32
  29. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
  30. 2023Retrieval-Augmented Text-to-Audio Generationcited by 24
  31. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  32. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  33. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generationcited by 23
  34. 2016High-Quality, Low-Delay Music Coding in the Opus Codeccited by 13
  35. 2024Neural Speech Coding for Real-time Communications using Constant Bitrate Scalar Quantizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 23.

49 papers.

Sort:

By area:

  1. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  2. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  4. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 38
  5. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  6. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 22
  7. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  8. 2025NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inferencecited by 15
  9. 2026MOSS-TTS Technical Reportcited by 14
  10. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 13
  11. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  12. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 9
37 more
  1. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  2. 2025CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audiocited by 7
  3. 2025SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codeccited by 5
  4. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  5. 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Codecited by 4
  6. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  7. 2025Next Tokens Denoising for Speech Synthesiscited by 3
  8. 2026SAME: A Semantically-Aligned Music Autoencodercited by 3
  9. 2025UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic informationcited by 2
  10. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  11. 2025Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-ratescited by 2
  12. 2026UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstructioncited by 2
  13. 2026Sylber 2.0: A Universal Syllable Embeddingcited by 2
  14. 2026Scaling Speech Tokenizers with Diffusion Autoencoderscited by 2
  15. 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 2
  16. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  17. 2025What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Studycited by 1
  18. 2025Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representationscited by 1
  19. 2025U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generationcited by 1
  20. 2026VP-VAE: Rethinking Vector Quantization via Adaptive Vector Perturbationcited by 1
  21. 2026Shao: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generationcited by 1
  22. 2026Exploring Token-Space Manipulation in Latent Audio Tokenizerscited by 1
  23. 2025Robust Residual Finite Scalar Quantization for Neural Compressioncited by 0
  24. 2025FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecscited by 0
  25. 2025Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditionscited by 0
  26. 2025Two-Dimensional Quantization for Geometry-Aware Audio Codingcited by 0
  27. 2026STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecscited by 0
  28. 2026From Fewer Samples to Fewer Bits: Reframing Dataset Distillation as Joint Optimization of Precision and Compactnesscited by 0
  29. 2026Voxtral TTScited by 0
  30. 2026CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encodingcited by 0
  31. 2026Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speechcited by 0
  32. 2026Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignmentcited by 0
  33. 2026ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Modelscited by 0
  34. 2026Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokenscited by 0
  35. 2026Combining Self-Embedding Audio Watermarking with Ultra-Low-Bitrate Neural Codecscited by 0
  36. 2026ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Codingcited by 0
  37. 2026GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Codingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Ultra-Low-Bitrate Speech Coding with Pretrained Transformers
  2. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  3. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference
  4. 2025Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
  5. 2026One-Step Token-to-Waveform Generation with MeanFlow in Latent Space ↗
  6. 2022Cross-Scale Vector Quantization for Scalable Neural Speech Coding
  7. 2026AudioTQ: A Data-Oblivious 6-Bit CPU Audio Codec via Randomized Hadamard Rotation and Lloyd-Max Quantization
  8. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling