mlsanityData to 19 Oct 2026

ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion

arXiv 2606.21888 ↗First posted 20 Jun 2026eess.AS cs.SDcited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Neural speech codecs efficiently compress speech and have become a foundation for speech generation, but they are typically learned as holistic representations that intertwine linguistic content, speaker identity, and prosody. While this design is effective for zero-shot voice cloning, it hinders downstream tasks that require prosody preservation or transfer, such as voice conversion. To address this, we introduce ProsoCodec, a prosody-oriented speech codec that models prosody as a conditional residual rather than as a disentangled stream. Specifically, by conditioning both the encoder and decoder on text and speaker embeddings as prefix tokens, the discrete bottleneck is encouraged to capture prosodic variation not explained by content and speaker. To further preserve prosody, we use the low-frequency mel band and train the model on paired same-speaker utterances. Experiments on voice conversion show improved prosody preservation and reduced source-timbre leakage.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 38 of its 43 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

38 of 43 references.

Closest ancestors · 18

Same direction, newest first

  1. 2026Scaling Speech Tokenizers with Diffusion Autoencoders6 Feb
  2. 2026A Unified Neural Codec Language Model for Selective Editable Text to Speech Generation18 Jan
  3. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling22 Aug
  4. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training23 May
  5. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis6 Feb
  6. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Coding29 Nov
  7. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  8. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling29 Aug
  9. 2024Autoregressive Speech Synthesis without Vector Quantization11 Jul
  10. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation11 Jun
  11. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Models4 Jun
  12. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild25 Mar
6 more
  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  2. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis21 Nov
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale23 Jun
  4. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias6 Jun
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  6. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech16 Feb

Other references · 20

  1. 2017Attention Is All You Needcited by 61,180
  2. 2022Scalable Diffusion Models with Transformerscited by 6,161
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  5. 2022High Fidelity Neural Audio Compressioncited by 917
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 864
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  8. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 287
  9. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 232
  10. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 218
  11. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 162
  12. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 162
8 more
  1. 2026Qwen3-ASR Technical Reportcited by 162
  2. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 118
  3. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 103
  4. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 100
  5. 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 68
  6. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 57
  7. 2025Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglementcited by 52
  8. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 44

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Prosody-Adaptable Audio Codecs for Zero-Shot Voice Conversion via In-Context Learning
  2. 2022Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech
  3. 2025ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs
  4. 2022Disentangling Prosody Representations with Unsupervised Speech Reconstruction
  5. 2023Towards General-Purpose Text-Instruction-Guided Voice Conversion
  6. 2024FreeCodec: A disentangled neural speech codec with fewer tokens
  7. 2021Prosody-TTS: An end-to-end speech synthesis system with prosody control
  8. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling