# Multi-speaker Emotional Text-to-speech Synthesizer

arXiv: https://arxiv.org/abs/2112.03557 · HTML: https://mlsanity.com/paper/2112.03557/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Emotional End-to-End Neural Speech Synthesizer](https://mlsanity.com/paper/1711.05447/) (2017)
- [Emotional speech synthesis with rich and granularized control](https://mlsanity.com/paper/1911.01635/) (2019)
- [Emotional Voice Conversion using Multitask Learning with Text-to-speech](https://mlsanity.com/paper/1911.06149/) (2019)

## Built on by (most cited first, 1 of 1)

- [EmoNews: A Spoken Dialogue System for Expressive News Conversations](https://mlsanity.com/paper/2506.13894/) (2025)

## Abstract (arXiv)

We present a methodology to train our multi-speaker emotional text-to-speech synthesizer that can express speech for 10 speakers' 7 different emotions. All silences from audio samples are removed prior to learning. This results in fast learning by our model. Curriculum learning is applied to train our model efficiently. Our model is first trained with a large single-speaker neutral dataset, and then trained with neutral speech from all speakers. Finally, our model is trained using datasets of emotional speech from all speakers. In each stage, training samples of each speaker-emotion pair have equal probability to appear in mini-batches. Through this procedure, our model can synthesize speech for all targeted speakers and emotions. Our synthesized audio sets are available on our web page.