mlsanityData to 19 Oct 2026

SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis

arXiv 2409.00933 ↗First posted 2 Sep 2024cs.SD eess.AScited by 13

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

The long speech sequence has been troubling language models (LM) based TTS approaches in terms of modeling complexity and efficiency. This work proposes SoCodec, a semantic-ordered multi-stream speech codec, to address this issue. It compresses speech into a shorter, multi-stream discrete semantic sequence with multiple tokens at each frame. Meanwhile, the ordered product quantization is proposed to constrain this sequence into an ordered representation. It can be applied with a multi-stream delayed LM to achieve better autoregressive generation along both time and stream axes in TTS. The experimental result strongly demonstrates the effectiveness of the proposed approach, achieving superior performance over baseline systems even if compressing the frameshift of speech from 20ms to 240ms (12x). The ablation studies further validate the importance of learning the proposed ordered multi-stream semantic representation in pursuing shorter speech sequences for efficient LM-based TTS.

Key facts. Cited by 13 papers in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 26 of its 31 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

26 of 31 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation11 Jun
  2. 2024Addressing Index Collapse of Large-Codebook Speech Tokenizer with Dual-Decoding Product-Quantized Variational Auto-Encoder5 Jun
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  4. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data12 Feb
  5. 2023Fewer-token Neural Speech Codec with Time-invariant Codes15 Sep
  6. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models31 Aug
  7. 2023Better speech synthesis through scaling12 May
  8. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec4 May
  9. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling7 Mar
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  11. 2021Text-Free Prosody-Aware Generative Spoken Language Modeling7 Sep

Other references · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,760
  2. 2023GPT-4 Technical Reportcited by 19,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,489
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  6. 2022High Fidelity Neural Audio Compressioncited by 917
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
  8. 2023Simple and Controllable Music Generationcited by 540
  9. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 315
  10. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 270
  11. 2014Learning Ordered Representations with Nested Dropoutcited by 87
  12. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 84
3 more
  1. 2021ECAPA-TDNN Embeddings for Speaker Diarizationcited by 60
  2. 2024WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmarkcited by 54
  3. 2021Anytime Sampling for Autoregressive Models via Ordered Autoencodingcited by 13

Built on by

Cited by year (2026 to 19 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 107
  2. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 106
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 65
  4. 2025CodecFake+: Codec-Based Resynthesized Data as a Proxy for Detecting CodecFake Speechcited by 9
  5. 2025FireRedTTS-1S: An Upgraded Streamable Foundation Text-to-Speech Systemcited by 9
  6. 2025CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidancecited by 7
  7. 2024Speaking from Coarse to Fine: Improving Neural Codec Language Model via Multi-Scale Speech Coding and Generationcited by 6
  8. 2025SpecTokenizer: A Lightweight Streaming Codec in the Compressed Spectrum Domaincited by 2
  9. 2026ContextCodec: Content-Focused Context Guidance for Ultra-Low Bitrate Speech Codingcited by 1
  10. 2026Benchmarking Neural Speech Compression from a Rate-Distortion Perspectivecited by 1
  11. 2025Parallel GPT: Harmonizing the Independence and Interdependence of Acoustic and Semantic Information for Zero-Shot Text-to-Speechcited by 0
  12. 2026LL-SDR: Low-Latency Speech enhancement through Discrete Representationscited by 0
1 more
  1. 2026Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Comprehend and Talk: Text to Speech Synthesis via Dual Language Modeling
  2. 2025UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information
  3. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech
  4. 2025SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization
  5. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
  6. 2026Rethinking Speech Codecs: From Compression to Autoregressive Generative Modeling
  7. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  8. 2024Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesis