# DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarin arXiv: https://arxiv.org/abs/2309.00883 ยท HTML: https://mlsanity.com/paper/2309.00883/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Audio, speech and music, one of 14 directions in Diffusion models. - Builds on 53 of its 74 references that are in the map. - First posted on arXiv 2 Sep 2023. ## Placed in - Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/) ## Builds on (most cited first, 30 of 53) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020) - [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022) - [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017) - [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020) - [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017) - [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021) - [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018) - [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018) - [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022) - [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018) - [Diff-TTS: A Denoising Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2104.01409/) (2021) - [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021) - [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022) - [FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis](https://mlsanity.com/paper/2204.09934/) (2022) - [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021) - [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021) - [Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance](https://mlsanity.com/paper/2111.11755/) (2021) - [PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior](https://mlsanity.com/paper/2106.06406/) (2021) - [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017) - [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019) - [WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis](https://mlsanity.com/paper/2106.09660/) (2021) - [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018) - [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018) ## Built on by (most cited first, 2 of 2) - [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024) - [Optimizing Multilingual Text-To-Speech with Accents & Emotions](https://mlsanity.com/paper/2506.16310/) (2025) ## Abstract (arXiv) While the performance of cross-lingual TTS based on monolingual corpora has been significantly improved recently, generating cross-lingual speech still suffers from the foreign accent problem, leading to limited naturalness. Besides, current cross-lingual methods ignore modeling emotion, which is indispensable paralinguistic information in speech delivery. In this paper, we propose DiCLET-TTS, a Diffusion model based Cross-Lingual Emotion Transfer method that can transfer emotion from a source speaker to the intra- and cross-lingual target speakers. Specifically, to relieve the foreign accent problem while improving the emotion expressiveness, the terminal distribution of the forward diffusion process is parameterized into a speaker-irrelevant but emotion-related linguistic prior by a prior text encoder with the emotion embedding as a condition. To address the weaker emotional expressiveness problem caused by speaker disentanglement in emotion embedding, a novel orthogonal projection based emotion disentangling module (OP-EDM) is proposed to learn the speaker-irrelevant but emotion-discriminative embedding. Moreover, a condition-enhanced DPM decoder is introduced to strengthen the modeling ability of the speaker and the emotion in the reverse diffusion process to further improve emotion expressiveness in speech delivery. Cross-lingual emotion transfer experiments show the superiority of DiCLET-TTS over various competitive models and the good design of OP-EDM in learning speaker-irrelevant but emotion-discriminative embedding.