mlsanityData to 7 Oct 2026

Fine-grained style control in Transformer-based Text-to-speech Synthesis

arXiv 2110.06306 ↗First posted 12 Oct 2021eess.AS cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel architecture to realize fine-grained style control on the transformer-based text-to-speech synthesis (TransformerTTS). Specifically, we model the speaking style by extracting a time sequence of local style tokens (LST) from the reference speech. The existing content encoder in TransformerTTS is then replaced by our designed cross-attention blocks for fusion and alignment between content and style. As the fusion is performed along with the skip connection, our cross-attention block provides a good inductive bias to gradually infuse the phoneme representation with a given style. Additionally, we prevent the style embedding from encoding linguistic content by randomly truncating LST during training and using wav2vec 2.0 features. Experiments show that with fine-grained style control, our system performs better in terms of naturalness, intelligibility, and style transferability. Our code and samples are publicly available.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 19 of its 27 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

19 of 27 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  7. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  8. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  9. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  10. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  11. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  12. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
7 more
  1. 2020MultiSpeech: Multi-Speaker Text to Speech with Transformercited by 42
  2. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
  3. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  4. 2020Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognitioncited by 19
  5. 2021Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminabilitycited by 19
  6. 2020Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesiscited by 11
  7. 2020Fine-grained Style Modeling, Transfer and Prediction in Text-to-Speech Synthesis via Phone-Level Content-Style Disentanglementcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 2; 2025: 4; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023Transformers in Speech Processing: A Surveycited by 38
  2. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  3. 2022Controllable Data Generation by Deep Learning: A Reviewcited by 20
  4. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  5. 2022Recent Progress in Transformer-based Medical Image Analysiscited by 9
  6. 2024ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesiscited by 9
  7. 2023Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesiscited by 7
  8. 2022A unified one-shot prosody and speaker conversion system with self-supervised discrete speech unitscited by 3
  9. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  10. 2024Towards Evaluating the Robustness of Automatic Speech Recognition Systems via Audio Style Transfercited by 2
  11. 2025Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimizationcited by 2
  12. 2023DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixercited by 1
3 more
  1. 2025HiStyle: Hierarchical Style Embedding Predictor for Text-Prompt-Guided Controllable Speech Synthesiscited by 1
  2. 2025MiSTR: Multi-Modal iEEG-to-Speech Synthesis with Transformer-Based Prosody Prediction and Neural Phase Reconstructioncited by 0
  3. 2026Brain-to-Speech: Prosody Feature Engineering and Transformer-Based Reconstructioncited by 0