mlsanityData to 14 Oct 2026

Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis

arXiv 2111.10177 ↗First posted 19 Nov 2021cs.SD cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper presents a method for controlling the prosody at the phoneme level in an autoregressive attention-based text-to-speech system. Instead of learning latent prosodic features with a variational framework as is commonly done, we directly extract phoneme-level F0 and duration features from the speech data in the training set. Each prosodic feature is discretized using unsupervised clustering in order to produce a sequence of prosodic labels for each utterance. This sequence is used in parallel to the phoneme sequence in order to condition the decoder with the utilization of a prosodic encoder and a corresponding attention module. Experimental results show that the proposed method retains the high quality of generated speech, while allowing phoneme-level control of F0 and duration. By replacing the F0 cluster centroids with musical notes, the model can also provide control over the note and octave within the range of the speaker.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 19 Nov 2021.

Builds on

24 of 30 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  3. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  4. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  5. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  6. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  7. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 110
  8. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  9. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  10. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  11. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  12. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
12 more
  1. 2019MelNet: A Generative Model for Audio in the Frequency Domaincited by 42
  2. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  3. 2020Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody priorcited by 33
  4. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  5. 2019CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Networkcited by 25
  6. 2019Semi-Supervised Generative Modeling for Controllable Speech Synthesiscited by 21
  7. 2020Controllable neural text-to-speech synthesis using intuitive prosodic featurescited by 19
  8. 2021High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latencycited by 19
  9. 2020Bunched LPCNet : Vocoder for Low-cost Neural Text-To-Speech Systemscited by 16
  10. 2019Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesiscited by 15
  11. 2019Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilitiescited by 11
  12. 2019Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Featurescited by 10

Built on by

Cited by year

Citing papers per year: 2022: 7; 2023: 1.

8 papers.

  1. 2022Unsupervised word-level prosody tagging for controllable speech synthesiscited by 7
  2. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  3. 2022Into-TTS : Intonation Template Based Prosody Control Systemcited by 2
  4. 2022A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesiscited by 2
  5. 2022Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Featurescited by 2
  6. 2022Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesiscited by 0
  7. 2022Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesiscited by 0
  8. 2023Using Deepfake Technologies for Word Emphasis Detectioncited by 0