# Cross-Speaker Emotion Transfer for Low-Resource Text-to-Speech Using Non-Parallel Voice Conversion with Pitch-Shift Data Augmentation

arXiv: https://arxiv.org/abs/2204.10020 · HTML: https://mlsanity.com/paper/2204.10020/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion](https://mlsanity.com/paper/1904.04631/) (2019)
- [Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens](https://mlsanity.com/paper/1910.11997/) (2019)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis](https://mlsanity.com/paper/2011.08477/) (2020)
- [Low-resource expressive text-to-speech using data augmentation](https://mlsanity.com/paper/2011.05707/) (2020)
- [Exploring Transfer Learning for Low Resource Emotional TTS](https://mlsanity.com/paper/1901.04276/) (2019)
- [TTS-by-TTS: TTS-driven Data Augmentation for Fast and High-Quality Speech Synthesis](https://mlsanity.com/paper/2010.13421/) (2020)
- [Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language](https://mlsanity.com/paper/1810.11960/) (2018)
- [Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech](https://mlsanity.com/paper/2106.12896/) (2021)
- [Cross-speaker style transfer for text-to-speech using data augmentation](https://mlsanity.com/paper/2202.05083/) (2022)

## Built on by (most cited first, 8 of 8)

- [ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models](https://mlsanity.com/paper/2305.13831/) (2023)
- [Nonparallel Emotional Voice Conversion For Unseen Speaker-Emotion Pairs Using Dual Domain Adversarial Network & Virtual Domain Pairing](https://mlsanity.com/paper/2302.10536/) (2023)
- [Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings](https://mlsanity.com/paper/2407.14056/) (2024)
- [CALLS: Japanese Empathetic Dialogue Speech Corpus of Complaint Handling and Attentive Listening in Customer Center](https://mlsanity.com/paper/2305.13713/) (2023)
- [Voice Conversion for Lombard Speaking Style with Implicit and Explicit Acoustic Feature Conditioning](https://mlsanity.com/paper/2507.09310/) (2025)
- [Unsupervised Pre-Training For Data-Efficient Text-to-Speech On Low Resource Languages](https://mlsanity.com/paper/2303.15669/) (2023)
- [Discrete Unit based Masking for Improving Disentanglement in Voice Conversion](https://mlsanity.com/paper/2409.11560/) (2024)
- [Semantics-Aware Generative Latent Data Augmentation for Learning in Low-Resource Domains](https://mlsanity.com/paper/2602.02841/) (2026)

## Abstract (arXiv)

Data augmentation via voice conversion (VC) has been successfully applied to low-resource expressive text-to-speech (TTS) when only neutral data for the target speaker are available. Although the quality of VC is crucial for this approach, it is challenging to learn a stable VC model because the amount of data is limited in low-resource scenarios, and highly expressive speech has large acoustic variety. To address this issue, we propose a novel data augmentation method that combines pitch-shifting and VC techniques. Because pitch-shift data augmentation enables the coverage of a variety of pitch dynamics, it greatly stabilizes training for both VC and TTS models, even when only 1,000 utterances of the target speaker's neutral data are available. Subjective test results showed that a FastSpeech 2-based emotional TTS system with the proposed method improved naturalness and emotional similarity compared with conventional methods.