DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarin
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
While the performance of cross-lingual TTS based on monolingual corpora has been significantly improved recently, generating cross-lingual speech still suffers from the foreign accent problem, leading to limited naturalness. Besides, current cross-lingual methods ignore modeling emotion, which is indispensable paralinguistic information in speech delivery. In this paper, we propose DiCLET-TTS, a Diffusion model based Cross-Lingual Emotion Transfer method that can transfer emotion from a source speaker to the intra- and cross-lingual target speakers. Specifically, to relieve the foreign accent problem while improving the emotion expressiveness, the terminal distribution of the forward diffusion process is parameterized into a speaker-irrelevant but emotion-related linguistic prior by a prior text encoder with the emotion embedding as a condition. To address the weaker emotional expressiveness problem caused by speaker disentanglement in emotion embedding, a novel orthogonal projection based emotion disentangling module (OP-EDM) is proposed to learn the speaker-irrelevant but emotion-discriminative embedding. Moreover, a condition-enhanced DPM decoder is introduced to strengthen the modeling ability of the speaker and the emotion in the reverse diffusion process to further improve emotion expressiveness in speech delivery. Cross-lingual emotion transfer experiments show the superiority of DiCLET-TTS over various competitive models and the good design of OP-EDM in learning speaker-irrelevant but emotion-discriminative embedding.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 53 of its 74 references that are in the map. First posted on arXiv 2 Sep 2023.
Builds on
53 of 74 references.
Closest ancestors · 10
Same direction, newest first
- 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
- 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
- 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data30 May
- 2022BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis30 May
- 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis21 Apr
- 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov
- 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme28 Sep
- 2021PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior11 Jun
- 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
- 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
Other references · 43
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2022Denoising Diffusion Restoration Modelscited by 951
- 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
- 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
- 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
31 more
- 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
- 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
- 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
- 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
- 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
- 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
- 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
- 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
- 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
- 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
- 2021Orthogonal Projection Losscited by 44
- 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
- 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
- 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
- 2020One Model, Many Languages: Meta-learning for Multilingual Text-to-Speechcited by 28
- 2020CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speechcited by 22
- 2021Cross-speaker emotion disentangling and transfer for end-to-end speech synthesiscited by 21
- 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
- 2022iEmoTTS: Toward Robust Cross-Speaker Emotion Transfer and Control for Speech Synthesis based on Disentanglement between Prosody and Timbrecited by 15
- 2019Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistencycited by 12
- 2020Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversioncited by 12
- 2019Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesiscited by 11
- 2021Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesiscited by 9
- 2021Cross-lingual Low Resource Speaker Adaptation Using Phonological Featurescited by 8
- 2022Learn2Sing 2.0: Diffusion and Mutual Information-Based Target Speaker SVS by Learning from Singing Teachercited by 8
- 2022Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesiscited by 8
- 2021Multi-speaker Multi-style Text-to-speech Synthesis With Single-speaker Single-style Training Data Scenarioscited by 5
- 2022Accurate Emotion Strength Assessment for Seen and Unseen Speech Based on Data-Driven Deep Learningcited by 5
- 2022Disentangling Style and Speaker Attributes for TTS Style Transfercited by 3
- 2021Exploring Timbre Disentanglement in Non-Autoregressive Cross-Lingual Text-to-Speechcited by 1
- 2022Improving Cross-lingual Speech Synthesis with Triplet Training Schemecited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025XEmoRAG: Cross-Lingual Emotion Transfer with Controllable Intensity Using Retrieval-Augmented Generation ↗
- 2023EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Data
- 2024Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Models
- 2024ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesis
- 2024Hierarchical Control of Emotion Rendering in Speech Synthesis
- 2023ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models
- 2025Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization
- 2026Acoustic and Semantic Modeling of Emotion in Spoken Language ↗