mlsanityData to 13 Oct 2026

Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control

arXiv 2111.10168 ↗First posted 19 Nov 2021cs.SD cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper presents a method for phoneme-level prosody control of F0 and duration on a multispeaker text-to-speech setup, which is based on prosodic clustering. An autoregressive attention-based model is used, incorporating multispeaker architecture modules in parallel to a prosody encoder. Several improvements over the basic single-speaker method are proposed that increase the prosodic control range and coverage. More specifically we employ data augmentation, F0 normalization, balanced clustering for duration, and speaker-independent prosodic clustering. These modifications enable fine-grained phoneme-level prosody control for all speakers contained in the training set, while maintaining the speaker identity. The model is also fine-tuned to unseen speakers with limited amounts of data and it is shown to maintain its prosody control capabilities, verifying that the speaker-independent prosodic clustering is effective. Experimental results verify that the model maintains high output speech quality and that the proposed method allows efficient prosody control within each speaker's range despite the variability that a multispeaker setting introduces.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 19 Nov 2021.

Builds on

27 of 39 references.

References · 27

  1. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 803
  2. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 489
  3. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  4. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 172
  5. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  6. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  7. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  8. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  9. 2019Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokenscited by 44
  10. 2020Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesiscited by 41
  11. 2020Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody priorcited by 33
  12. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
15 more
  1. 2019CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Networkcited by 25
  2. 2020CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speechcited by 22
  3. 2020Controllable neural text-to-speech synthesis using intuitive prosodic featurescited by 19
  4. 2021High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latencycited by 19
  5. 2020Bunched LPCNet : Vocoder for Low-cost Neural Text-To-Speech Systemscited by 16
  6. 2020Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesiscited by 16
  7. 2019Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesiscited by 15
  8. 2021Adversarially learning disentangled speech representations for robust multi-factor voice conversioncited by 14
  9. 2019Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilitiescited by 11
  10. 2021Expressive Neural Voice Cloningcited by 11
  11. 2019Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Featurescited by 10
  12. 2020Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfercited by 10
  13. 2020Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS?cited by 9
  14. 2019Singing Synthesis: with a little help from my attentioncited by 8
  15. 2020Few Shot Adaptive Normalization Driven Multi-Speaker Speech Synthesiscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1.

3 papers.

  1. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  2. 2022Karaoker: Alignment-free singing voice synthesis with speech training datacited by 1
  3. 2024PRESENT: Zero-Shot Text-to-Prosody Controlcited by 0