# METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer

arXiv: https://arxiv.org/abs/2307.15951 · HTML: https://mlsanity.com/paper/2307.15951/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2023.

## Built on by (most cited first, 6 of 6)

- [UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts](https://mlsanity.com/paper/2404.18398/) (2024)
- [DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech](https://mlsanity.com/paper/2505.19687/) (2025)
- [ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training](https://mlsanity.com/paper/2501.04416/) (2025)
- [Beyond Global Emotion: Fine-Grained Emotional Speech Synthesis with Dynamic Word-Level Modulation](https://mlsanity.com/paper/2509.20378/) (2025)
- [GTR-Voice: Articulatory Phonetics Informed Controllable Expressive Speech Synthesis](https://mlsanity.com/paper/2406.10514/) (2024)
- [Optimizing Multilingual Text-To-Speech with Accents & Emotions](https://mlsanity.com/paper/2506.16310/) (2025)

## Abstract (arXiv)

Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer - the heavy entanglement of speaker timbre, emotion, and language factors in the speech signal will make a system produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes the Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift-based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization-based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.