mlsanityData to 7 Oct 2026

Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTS

arXiv 2207.06000 ↗First posted 13 Jul 2022cs.CL cs.LG cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

Expressive text-to-speech has shown improved performance in recent years. However, the style control of synthetic speech is often restricted to discrete emotion categories and requires training data recorded by the target speaker in the target style. In many practical situations, users may not have reference speech recorded in target emotion but still be interested in controlling speech style just by typing text description of desired emotional style. In this paper, we propose a text-based interface for emotional style control and cross-speaker style transfer in multi-speaker TTS. We propose the bi-modal style encoder which models the semantic relationship between text description embedding and speech style embedding with a pretrained language model. To further improve cross-speaker style transfer on disjoint, multi-style datasets, we propose the novel style loss. The experimental results show that our model can generate high-quality expressive speech even in unseen style.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 26 references that are in the map. First posted on arXiv 13 Jul 2022.

Builds on

23 of 26 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  9. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  10. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  11. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  12. 2020Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesiscited by 54
11 more
  1. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  2. 2021One TTS Alignment To Rule Them Allcited by 47
  3. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  4. 2017Emotional End-to-End Neural Speech Synthesizercited by 30
  5. 2022MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesiscited by 29
  6. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
  7. 2021Expressive Text-to-Speech using Style Tagcited by 20
  8. 2020Unified Mandarin TTS Front-end Based on Distilled BERT Modelcited by 18
  9. 2019Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistencycited by 12
  10. 2019Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesiscited by 11
  11. 2021Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesiscited by 9

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5.

6 papers.

  1. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  2. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 23
  3. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  4. 2024Controlling Emotion in Text-to-Speech with Natural Language Promptscited by 8
  5. 2024Exploring speech style spaces with language models: Emotional TTS without emotion labelscited by 4
  6. 2023Cross-speaker Emotion Transfer by Manipulating Speech Style Latentscited by 2