mlsanityData to 14 Oct 2026

Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech

arXiv 2206.12229 ↗First posted 24 Jun 2022cs.SD cs.CL eess.AScited by 13

Not on the map. Why?

Abstract (arXiv)

The cloning of a speaker's voice using an untranscribed reference sample is one of the great advances of modern neural text-to-speech (TTS) methods. Approaches for mimicking the prosody of a transcribed reference audio have also been proposed recently. In this work, we bring these two tasks together for the first time through utterance level normalization in conjunction with an utterance level speaker embedding. We further introduce a lightweight aligner for extracting fine-grained prosodic features, that can be finetuned on individual samples within seconds. We show that it is possible to clone the voice of a speaker as well as the prosody of a spoken reference independently without any degradation in quality and high similarity to both original voice and prosody, as our objective evaluation and human study show. All of our code and trained models are available, alongside static and interactive demos.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 24 Jun 2022.

Builds on

33 of 53 references.

References · 33

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  7. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  9. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  10. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  11. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  12. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
21 more
  1. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  2. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  3. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
  4. 2018Neural Voice Cloning with a Few Samplescited by 100
  5. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  6. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  7. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  8. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
  9. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  10. 2021The VoicePrivacy 2020 Challenge: Results and findingscited by 46
  11. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  12. 2020Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embeddingcited by 27
  13. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  14. 2022Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Featurescited by 20
  15. 2020Controllable neural text-to-speech synthesis using intuitive prosodic featurescited by 19
  16. 2021HUI-Audio-Corpus-German: A high quality TTS datasetcited by 15
  17. 2019Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilitiescited by 11
  18. 2021Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesiscited by 8
  19. 2019IMS-Speech: A Speech to Text Toolcited by 4
  20. 2021ADEPT: A Dataset for Evaluating Prosody Transfercited by 4
  21. 2021VRAIN-UPV MLLP's system for the Blizzard Challenge 2021cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 4; 2025: 3; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scalecited by 91
  2. 2022Low-Resource Multilingual and Zero-Shot Multispeaker TTScited by 15
  3. 2024Meta Learning Text-to-Speech Synthesis in over 7000 Languagescited by 9
  4. 2024Controlling Emotion in Text-to-Speech with Natural Language Promptscited by 9
  5. 2023The IMS Toucan System for the Blizzard Challenge 2023cited by 6
  6. 2022PoeticTTS -- Controllable Poetry Reading for Literary Studiescited by 5
  7. 2024Probing the Feasibility of Multilingual Speaker Anonymizationcited by 5
  8. 2025Voice Cloning: Comprehensive Surveycited by 4
  9. 2025Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Modelscited by 3
  10. 2025Investigating Stochastic Methods for Prosody Modeling in Speech Synthesiscited by 2
  11. 2023Improved Child Text-to-Speech Synthesis through Fastpitch-based Transfer Learningcited by 0
  12. 2026What You Train Is What You Get: Gender Bias, Training Composition, and Post-Hoc Mitigation in Audio Deepfake Detectioncited by 0
1 more
  1. 2026Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbingcited by 0