mlsanityData to 7 Oct 2026

Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron

arXiv 1803.09047 ↗First posted 24 Mar 2018cs.CL cs.LG cs.SDcited by 171

Not on the map. Why?

Abstract (arXiv)

We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 37; 2023: 33; 2024: 18; 2025: 19; 2026: 13.

171 papers.

Sort:

By area:

  1. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  2. 2021A Survey on Neural Speech Synthesiscited by 182
  3. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  4. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  5. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  6. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  7. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  8. 2021Half-Truth: A Partially Fake Audio Detection Datasetcited by 44
  9. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  10. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  11. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  12. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 32
38 more
  1. 2022An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Eracited by 32
  2. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speechcited by 30
  3. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  4. 2022Speech Synthesis with Mixed Emotionscited by 28
  5. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  6. 2021V2C: Visual Voice Cloningcited by 23
  7. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  8. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  9. 2023StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesiscited by 21
  10. 2021Multimodal Emotion Recognition with High-level Speech and Text Featurescited by 20
  11. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  12. 2021Towards Multi-Scale Style Control for Expressive Speech Synthesiscited by 18
  13. 2022SpeechPainter: Text-conditioned Speech Inpaintingcited by 17
  14. 2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
  15. 2021Fine-grained style control in Transformer-based Text-to-speech Synthesiscited by 15
  16. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Trainingcited by 15
  17. 2021Adversarially learning disentangled speech representations for robust multi-factor voice conversioncited by 14
  18. 2021NWT: Towards natural audio-to-video generation with representation learningcited by 14
  19. 2021Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speechcited by 14
  20. 2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
  21. 2021KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Datasetcited by 13
  22. 2021Speaker Generationcited by 13
  23. 2022Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speechcited by 13
  24. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  25. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
  26. 2021FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesiscited by 12
  27. 2021Emotional Prosody Control for Speech Generationcited by 12
  28. 2021More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speechcited by 12
  29. 2023Let There Be Sound: Reconstructing High Quality Speech from Silent Videoscited by 12
  30. 2021Expressive Neural Voice Cloningcited by 11
  31. 2021Style Equalization: Unsupervised Learning of Controllable Generative Sequence Modelscited by 11
  32. 2022Self-supervised Context-aware Style Representation for Expressive Speech Synthesiscited by 10
  33. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
  34. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  35. 2021Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesiscited by 9
  36. 2021Hierarchical prosody modeling and control in non-autoregressive parallel neural TTScited by 9
  37. 2021Emphasis control for parallel neural TTScited by 9
  38. 2022Cross-speaker style transfer for text-to-speech using data augmentationcited by 9

50 most cited of 171 All 171 (JSON)