mlsanityData to 19 Oct 2026

MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement

arXiv 2509.13068 ↗First posted 16 Sep 2025eess.AScited by 1

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Audio codecs are a critical component of modern speech generation systems. This paper introduces a low-bitrate, multi-scale residual codec that encodes speech into four distinct streams: semantic, timbre, prosody, and residual. This architecture achieves high-fidelity speech reconstruction at competitive low bitrates while demonstrating an inherent ability for information disentanglement. We construct a two-stage language model for text-to-speech (TTS) synthesis using this codec, which, despite its lightweight design and minimal data requirements, achieves a state-of-the-art Word Error Rate (WER) and superior speaker similarity compared to several larger models. Furthermore, the codec's design proves highly effective for voice conversion, enabling independent manipulation of speaker timbre and prosody. Our inference code, pre-trained models, and audio samples are available at https://github.com/herbertLJY/MSRCodec.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 31 of its 35 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

31 of 35 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis6 Feb
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
  3. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications5 Sep
  4. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens7 Jul
  5. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation11 Jun
  6. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers8 Jun
  7. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  8. 2023Fewer-token Neural Speech Codec with Time-invariant Codes15 Sep
  9. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models28 Aug
  10. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding13 Jun
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  12. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 19

  1. 2017Attention Is All You Needcited by 61,180
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 1,166
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 985
  5. 2022High Fidelity Neural Audio Compressioncited by 917
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 864
  7. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 589
  8. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  9. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 497
  10. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 270
  11. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 181
  12. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 162
7 more
  1. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 125
  2. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 84
  3. 2020CN-Celeb: multi-genre speaker recognitioncited by 66
  4. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 61
  5. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 57
  6. 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 30
  7. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 21

Built on by

Cited by year (2026 to 19 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026One-Step Token-to-Waveform Generation with MeanFlow in Latent Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AugCodec: A Low-Bitrate Disentangled Neural Speech Codec via Data Augmentation
  2. 2024Language-Codec: Bridging Discrete Codec Representations and Speech Language Models
  3. 2025A Novel Semantic Compression Approach for Ultra-low Bandwidth Voice Communication
  4. 2026Entropy-Guided GRVQ for Ultra-Low Bitrate Neural Speech Codec
  5. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  6. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs
  7. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding
  8. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks