mlsanityData to 14 Oct 2026

Towards Multi-Scale Style Control for Expressive Speech Synthesis

arXiv 2104.03521 ↗First posted 8 Apr 2021cs.SD cs.CL eess.AScited by 18

Not on the map. Why?

Abstract (arXiv)

This paper introduces a multi-scale speech style modeling method for end-to-end expressive speech synthesis. The proposed method employs a multi-scale reference encoder to extract both the global-scale utterance-level and the local-scale quasi-phoneme-level style features of the target speech, which are then fed into the speech synthesis model as an extension to the input phoneme sequence. During training time, the multi-scale style model could be jointly trained with the speech synthesis model in an end-to-end fashion. By applying the proposed method to style transfer task, experimental results indicate that the controllability of the multi-scale speech style model and the expressiveness of the synthesized speech are greatly improved. Moreover, by assigning different reference speeches to extraction of style on each scale, the flexibility of the proposed method is further revealed.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 8 Apr 2021.

Builds on

12 of 17 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  7. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  8. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  9. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  10. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  11. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  12. 2019End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Trainingcited by 19

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 4; 2024: 5.

18 papers.

Sort:

By area:

  1. 2021A Survey on Neural Speech Synthesiscited by 184
  2. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  3. 2022Speech Synthesis with Mixed Emotionscited by 28
  4. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  5. 2023StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesiscited by 21
  6. 2024StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbingcited by 21
  7. 2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
  8. 2022STUDIES: Corpus of Japanese Empathetic Dialogue Speech Towards Friendly Voice Agentcited by 9
  9. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTScited by 8
  10. 2024Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settingscited by 7
  11. 2024A Review of Human Emotion Synthesis Based on Generative Technologycited by 7
  12. 2023Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Modelcited by 4
6 more
  1. 2022MAST: Multiscale Audio Spectrogram Transformerscited by 3
  2. 2024Towards Expressive Video Dubbing with Multiscale Multimodal Context Interactioncited by 3
  3. 2023A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysiscited by 1
  4. 2024FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistencycited by 1
  5. 2022Towards Cross-speaker Reading Style Transfer on Audiobook Datasetcited by 0
  6. 2023CASEIN: Cascading Explicit and Implicit Control for Fine-grained Emotion Intensity Regulationcited by 0