mlsanityData to 7 Oct 2026

CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation

arXiv 2412.20048 ↗First posted 28 Dec 2024eess.AS cs.AI cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

The goal of this work is to generate natural speech in multiple languages while maintaining the same speaker identity, a task known as cross-lingual speech synthesis. A key challenge of cross-lingual speech synthesis is the language-speaker entanglement problem, which causes the quality of cross-lingual systems to lag behind that of intra-lingual systems. In this paper, we propose CrossSpeech++, which effectively disentangles language and speaker information and significantly improves the quality of cross-lingual speech synthesis. To this end, we break the complex speech generation pipeline into two simple components: language-dependent and speaker-dependent generators. The language-dependent generator produces linguistic variations that are not biased by specific speaker attributes. The speaker-dependent generator models acoustic variations that characterize speaker identity. By handling each type of information in separate modules, our method can effectively disentangle language and speaker representation. We conduct extensive experiments using various metrics, and demonstrate that CrossSpeech++ achieves significant improvements in cross-lingual speech synthesis, outperforming existing methods by a large margin.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 75 references that are in the map. First posted on arXiv 28 Dec 2024.

Builds on

48 of 75 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  9. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  10. 2016Language Modeling with Gated Convolutional Networkscited by 688
  11. 2021Domain Generalization with MixStylecited by 563
  12. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
36 more
  1. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  2. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  3. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  4. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  5. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  6. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  7. 2019Feature-Critic Networks for Heterogeneous Domain Generalizationcited by 130
  8. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  9. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  10. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  11. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  12. 2017JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesiscited by 57
  13. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  14. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  15. 2019Learning Disentangled Representations via Mutual Information Estimationcited by 52
  16. 2021One TTS Alignment To Rule Them Allcited by 47
  17. 2021Style Normalization and Restitution for Domain Generalization and Adaptationcited by 46
  18. 2020MultiSpeech: Multi-Speaker Text to Speech with Transformercited by 42
  19. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38
  20. 2021LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Searchcited by 27
  21. 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 24
  22. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
  23. 2020Phonological Features for 0-shot Multilingual Speech Synthesiscited by 20
  24. 2022Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Featurescited by 20
  25. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Trainingcited by 15
  26. 2023ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representationscited by 15
  27. 2020VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamicscited by 14
  28. 2023Let There Be Sound: Reconstructing High Quality Speech from Silent Videoscited by 12
  29. 2024FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocodercited by 11
  30. 2022Distilling a Pretrained Language Model to a Multilingual ASR Modelcited by 7
  31. 2024USAT: A Universal Speaker-Adaptive Text-to-Speech Approachcited by 6
  32. 2023DSE-TTS: Dual Speaker Embedding for Cross-Lingual Text-to-Speechcited by 5
  33. 2022Disentangled representation learning for multilingual speaker recognitioncited by 4
  34. 2023CrossSpeech: Speaker-independent Acoustic Representation for Cross-lingual Speech Synthesiscited by 3
  35. 2023Joint unsupervised and supervised learning for context-aware language identificationcited by 1
  36. 2023GenerTTS: Pronunciation Disentanglement for Timbre and Style Generalization in Cross-Lingual Text-to-Speechcited by 1

Built on by

Not cited yet.