mlsanityData to 7 Oct 2026

Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis

arXiv 2211.01327 ↗First posted 2 Nov 2022cs.SD cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A large part of the expressive speech synthesis literature focuses on learning prosodic representations of the speech signal which are then modeled by a prior distribution during inference. In this paper, we compare different prior architectures at the task of predicting phoneme level prosodic representations extracted with an unsupervised FVAE model. We use both subjective and objective metrics to show that normalizing flow based prior networks can result in more expressive speech at the cost of a slight drop in quality. Furthermore, we show that the synthesized speech has higher variability, for a given text, due to the nature of normalizing flows. We also propose a Dynamical VAE model, that can generate higher quality speech although with decreased expressiveness and variability compared to the flow based models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

27 of 32 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  4. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  5. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  6. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  7. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  8. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
  9. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  10. 2020Dynamical Variational Autoencoders: A Comprehensive Reviewcited by 127
  11. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  12. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
15 more
  1. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  2. 2014Learning Stochastic Recurrent Networkscited by 60
  3. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  4. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  5. 2020VFlow: More Expressive Generative Flows with Variational Data Augmentationcited by 38
  6. 2020Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody priorcited by 33
  7. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 31
  8. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
  9. 2021High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latencycited by 19
  10. 2020Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesiscited by 16
  11. 2019Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Featurescited by 10
  12. 2021Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesiscited by 8
  13. 2022Unsupervised word-level prosody tagging for controllable speech synthesiscited by 7
  14. 2022Expressive, Variable, and Controllable Duration Modelling in TTScited by 7
  15. 2021Word-Level Style Control for Expressive, Non-attentive Speech Synthesiscited by 5

Built on by

Not cited yet.