mlsanityData to 7 Oct 2026

Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis

arXiv 2002.03785 ↗First posted 6 Feb 2020eess.AS cs.LG cs.SDcited by 41

Not on the map. Why?

Abstract (arXiv)

This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model. It achieves multi-resolution modeling of prosody by conditioning finer level representations on coarser level ones. Additionally, it imposes hierarchical conditioning across all latent dimensions using a conditional variational auto-encoder (VAE) with an auto-regressive structure. Evaluation of reconstruction performance illustrates that the new structure does not degrade the model while allowing better interpretability. Interpretations of prosody attributes are provided together with the comparison between word-level and phone-level prosody representations. Moreover, both qualitative and quantitative evaluations are used to demonstrate the improvement in the disentanglement of the latent dimensions.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 16; 2022: 13; 2023: 7; 2024: 4; 2025: 1.

41 papers.

Sort:

By area:

  1. 2021A Survey on Neural Speech Synthesiscited by 182
  2. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  3. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 32
  4. 2022Speech Synthesis with Mixed Emotionscited by 28
  5. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  6. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  7. 2021STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speechcited by 18
  8. 2021FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesiscited by 12
  9. 2023Let There Be Sound: Reconstructing High Quality Speech from Silent Videoscited by 12
  10. 2021Style Equalization: Unsupervised Learning of Controllable Generative Sequence Modelscited by 11
  11. 2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
  12. 2021Hierarchical disentangled representation learning for singing voice conversioncited by 9
29 more
  1. 2021Hierarchical prosody modeling and control in non-autoregressive parallel neural TTScited by 9
  2. 2021Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesiscited by 8
  3. 2021Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesiscited by 8
  4. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTScited by 8
  5. 2022Unsupervised word-level prosody tagging for controllable speech synthesiscited by 7
  6. 2021Review of end-to-end speech synthesis technology based on deep learningcited by 6
  7. 2023DSVAE: Interpretable Disentangled Representation for Synthetic Speech Detectioncited by 6
  8. 2024Style Mixture of Experts for Expressive Text-To-Speech Synthesiscited by 6
  9. 2021Word-Level Style Control for Expressive, Non-attentive Speech Synthesiscited by 5
  10. 2021A learned conditional prior for the VAE acoustic space of a TTS systemcited by 4
  11. 2021Discrete Acoustic Space for an Efficient Sampling in Neural Text-To-Speechcited by 3
  12. 2021Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Controlcited by 3
  13. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  14. 2021Learning Robust Latent Representations for Controllable Speech Synthesiscited by 2
  15. 2022Fine-grained Noise Control for Multispeaker Speech Synthesiscited by 2
  16. 2022Controllable Accented Text-to-Speech Synthesiscited by 2
  17. 2021UniTTS: Residual Learning of Unified Embedding Space for Speech Style Controlcited by 1
  18. 2022ReCAB-VAE: Gumbel-Softmax Variational Inference Based on Analytic Divergencecited by 1
  19. 2022Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speechcited by 1
  20. 2023Controllable Prosody Generation With Partial Inputscited by 1
  21. 2023Cross-Utterance Conditioned VAE for Speech Generationcited by 1
  22. 2023A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysiscited by 1
  23. 2024Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modelingcited by 1
  24. 2022Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesiscited by 0
  25. 2022Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesiscited by 0
  26. 2023DPP-TTS: Diversifying prosodic features of speech via determinantal point processescited by 0
  27. 2024PRESENT: Zero-Shot Text-to-Prosody Controlcited by 0
  28. 2024Disentangling Genotype and Environment Specific Latent Features for Improved Trait Prediction using a Compositional Autoencodercited by 0
  29. 2025A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translationcited by 0