mlsanityData to 13 Oct 2026

DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixer

arXiv 2305.19567 ↗First posted 31 May 2023cs.SD cs.CL eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

Despite the huge successes made in neutral TTS, content-leakage remains a challenge. In this paper, we propose a new input representation and simple architecture to achieve improved prosody modeling. Inspired by the recent success in the use of discrete code in TTS, we introduce discrete code to the input of the reference encoder. Specifically, we leverage the vector quantizer from the audio compression model to exploit the diverse acoustic information it has already been trained on. In addition, we apply the modified MLP-Mixer to the reference encoder, making the architecture lighter. As a result, we train the prosody transfer TTS in an end-to-end manner. We prove the effectiveness of our method through both subjective and objective evaluations. We demonstrate that the reference encoder learns better speaker-independent prosody when discrete code is utilized as input in the experiments. In addition, we obtain comparable results even when fewer parameters are inputted.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

27 of 32 references.

References · 27

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,352
  4. 2015Variational Inference with Normalizing Flowscited by 2,163
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,442
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,148
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 762
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 745
  9. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 715
  10. 2022High Fidelity Neural Audio Compressioncited by 680
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 613
  12. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 559
15 more
  1. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 489
  2. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  3. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 232
  4. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  5. 2020End-to-End Adversarial Text-to-Speechcited by 77
  6. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesiscited by 64
  7. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  8. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
  9. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  10. 2022JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speechcited by 26
  11. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  12. 2021Fine-grained style control in Transformer-based Text-to-speech Synthesiscited by 15
  13. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoderscited by 15
  14. 2020Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesiscited by 11
  15. 2021Mixer-TTS: non-autoregressive, fast and compact text-to-speech model conditioned on language model embeddingscited by 8

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS ↗
  2. 2024Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling
  3. 2025Discrete Audio Tokens: More Than a Survey!
  4. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
  5. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech
  6. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling
  7. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
  8. 2026EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement