mlsanityData to 7 Oct 2026

LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec

arXiv 2410.15764 ↗First posted 21 Oct 2024eess.AS cs.AI cs.SDcited by 7

Not on the map. Why?

Abstract (arXiv)

Although discrete speech tokens have exhibited strong potential for language model-based speech generation, their high bitrates and redundant timbre information restrict the development of such models. In this work, we propose LSCodec, a discrete speech codec that has both low bitrate and speaker decoupling ability. LSCodec adopts a multi-stage unsupervised training framework with a speaker perturbation technique. A continuous information bottleneck is first established, followed by vector quantization that produces a discrete speaker-decoupled space. A discrete token vocoder finally refines acoustic details from LSCodec. By reconstruction evaluations, LSCodec demonstrates superior intelligibility and audio quality with only a single codebook and smaller vocabulary size than baselines. Voice conversion and speaker probing experiments prove the excellent speaker disentanglement of LSCodec, and ablation study verifies the effectiveness of the proposed training framework.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

36 of 43 references.

References · 36

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  8. 2022High Fidelity Neural Audio Compressioncited by 673
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
24 more
  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  2. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  3. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  4. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  5. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  6. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  7. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  8. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  9. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  10. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  11. 2022SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottleneckscited by 27
  12. 2023Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTScited by 24
  13. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generationcited by 23
  14. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  15. 2024UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learnercited by 22
  16. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  17. 2023Fewer-token Neural Speech Codec with Time-invariant Codescited by 21
  18. 2023SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attentioncited by 19
  19. 2023Acoustic BPE for Speech Generation with Discrete Tokenscited by 15
  20. 2022CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representationscited by 11
  21. 2022Disentangled Feature Learning for Real-Time Neural Speech Codingcited by 10
  22. 2024MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Modelcited by 9
  23. 2024Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representationscited by 7
  24. 2024On the Effectiveness of Acoustic BPE in Decoder-Only TTScited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  3. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  4. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  5. 2025Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extractioncited by 1
  6. 2026MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corporacited by 0
  7. 2026Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignmentcited by 0