mlsanityData to 7 Oct 2026

Learning latent representations for style control and transfer in end-to-end speech synthesis

arXiv 1812.04342 ↗First posted 11 Dec 2018cs.CL cs.SD eess.AScited by 61

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce the Variational Autoencoder (VAE) to an end-to-end speech synthesis model, to learn the latent representation of speaking styles in an unsupervised manner. The style representation learned through VAE shows good properties such as disentangling, scaling, and combination, which makes it easy for style control. Style transfer can be achieved in this framework by first inferring style representation through the recognition network of VAE, then feeding it into TTS network to guide the style in synthesizing speech. To avoid Kullback-Leibler (KL) divergence collapse in training, several techniques are adopted. Finally, the proposed model shows good performance of style control and outperforms Global Style Token (GST) model in ABX preference tests on style transfer.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 19; 2023: 10; 2024: 5; 2025: 2; 2026: 4.

61 papers.

Sort:

By area:

  1. 2021A Survey on Neural Speech Synthesiscited by 182
  2. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  3. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 32
  4. 2022Speech Synthesis with Mixed Emotionscited by 28
  5. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  6. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  7. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  8. 2021STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speechcited by 18
  9. 2024EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Controlcited by 13
  10. 2021FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesiscited by 12
  11. 2021More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speechcited by 12
  12. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
38 more
  1. 2021Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesiscited by 9
  2. 2021Hierarchical prosody modeling and control in non-autoregressive parallel neural TTScited by 9
  3. 2022Cross-speaker style transfer for text-to-speech using data augmentationcited by 9
  4. 2021Learning De-identified Representations of Prosody from Raw Audiocited by 8
  5. 2021Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesiscited by 8
  6. 2022Cross-Speaker Emotion Transfer for Low-Resource Text-to-Speech Using Non-Parallel Voice Conversion with Pitch-Shift Data Augmentationcited by 8
  7. 2022Disentangling Prosody Representations with Unsupervised Speech Reconstructioncited by 8
  8. 2022Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesiscited by 7
  9. 2023Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesiscited by 7
  10. 2025Model See Model Do: Speech-Driven Facial Animation with Style Controlcited by 7
  11. 2021Review of end-to-end speech synthesis technology based on deep learningcited by 6
  12. 2023DSVAE: Interpretable Disentangled Representation for Synthetic Speech Detectioncited by 6
  13. 2023SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizercited by 6
  14. 2024Improving Generalization for AI-Synthesized Voice Detectioncited by 6
  15. 2021Word-Level Style Control for Expressive, Non-attentive Speech Synthesiscited by 5
  16. 2022PoeticTTS -- Controllable Poetry Reading for Literary Studiescited by 5
  17. 2023Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representationscited by 5
  18. 2021A learned conditional prior for the VAE acoustic space of a TTS systemcited by 4
  19. 2023Do Prosody Transfer Models Transfer Prosody?cited by 4
  20. 2021Enhancing Word-Level Semantic Representation via Dependency Structure for Expressive Text-to-Speech Synthesiscited by 3
  21. 2021On Prosody Modeling for ASR+TTS based Voice Conversioncited by 3
  22. 2021Discrete Acoustic Space for an Efficient Sampling in Neural Text-To-Speechcited by 3
  23. 2021Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Controlcited by 3
  24. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  25. 2022Emotion Selectable End-to-End Text-based Speech Editingcited by 3
  26. 2024Disentangling segmental and prosodic factors to non-native speech comprehensibilitycited by 3
  27. 2021Energy Disaggregation using Variational Autoencoderscited by 2
  28. 2021Learning Robust Latent Representations for Controllable Speech Synthesiscited by 2
  29. 2022Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesiscited by 2
  30. 2022A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesiscited by 2
  31. 2022Accented Text-to-Speech Synthesis with a Conditional Variational Autoencodercited by 2
  32. 2023An investigation of phrase break prediction in an End-to-End TTS systemcited by 2
  33. 2023DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarincited by 2
  34. 2021Using multiple reference audios and style embedding constraints for speech synthesiscited by 1
  35. 2021DeepA: A Deep Neural Analyzer For Speech And Singing Vocodingcited by 1
  36. 2022Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesiscited by 1
  37. 2022ReCAB-VAE: Gumbel-Softmax Variational Inference Based on Analytic Divergencecited by 1
  38. 2023Expressive TTS Driven by Natural Language Prompts Using Few Human Annotationscited by 1

50 most cited of 61 All 61 (JSON)