mlsanityData to 14 Oct 2026

Hierarchical prosody modeling and control in non-autoregressive parallel neural TTS

arXiv 2110.02952 ↗First posted 6 Oct 2021eess.AS cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Neural text-to-speech (TTS) synthesis can generate speech that is indistinguishable from natural speech. However, the synthetic speech often represents the average prosodic style of the database instead of having more versatile prosodic variation. Moreover, many models lack the ability to control the output prosody, which does not allow for different styles for the same text input. In this work, we train a non-autoregressive parallel neural TTS front-end model hierarchically conditioned on both coarse and fine-grained acoustic speech features to learn a latent prosody space with intuitive and meaningful dimensions. Experiments show that a non-autoregressive TTS model hierarchically conditioned on utterance-wise pitch, pitch range, duration, energy, and spectral tilt can effectively control each prosodic dimension, generate a wide variety of speaking styles, and provide word-wise emphasis control, while maintaining equal or better quality to the baseline model.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

22 of 31 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  5. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  6. 2018Efficient Neural Audio Synthesiscited by 219
  7. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  8. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  9. 2020End-to-End Adversarial Text-to-Speechcited by 77
  10. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  11. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  12. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
10 more
  1. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  2. 2019CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Networkcited by 25
  3. 2020Controllable neural text-to-speech synthesis using intuitive prosodic featurescited by 19
  4. 2020Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesiscited by 16
  5. 2018Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent languagecited by 14
  6. 2019Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilitiescited by 11
  7. 2020Prosodic Prominence and Boundaries in Sequence-to-Sequence Speech Synthesiscited by 10
  8. 2021On-device neural speech synthesiscited by 9
  9. 2021Supervised and Unsupervised Approaches for Controlling Narrow Lexical Focus in Sequence-to-Sequence Speech Synthesiscited by 8
  10. 2021Hierarchical Context-Aware Transformers for Non-Autoregressive Text to Speechcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 2; 2025: 2.

9 papers.

  1. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Trainingcited by 16
  2. 2022Prosody-controllable spontaneous TTS with neural HMMscited by 7
  3. 2023ADAPTERMIX: Exploring the Efficacy of Mixture of Adapters for Low-Resource TTS Adaptationcited by 7
  4. 2024Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modelingcited by 6
  5. 2025Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Modelscited by 3
  6. 2025CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidancecited by 3
  7. 2022Controllable Accented Text-to-Speech Synthesiscited by 2
  8. 2022Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesiscited by 0
  9. 2024PRODIS -- a speech database and a phoneme-based language model for the study of predictability effects in Polishcited by 0