# CrossSpeech: Speaker-independent Acoustic Representation for Cross-lingual Speech Synthesis

arXiv: https://arxiv.org/abs/2302.14370 · HTML: https://mlsanity.com/paper/2302.14370/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Feature-Critic Networks for Heterogeneous Domain Generalization](https://mlsanity.com/paper/1901.11448/) (2019)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [Pushing the limits of raw waveform speaker recognition](https://mlsanity.com/paper/2203.08488/) (2022)
- [One TTS Alignment To Rule Them All](https://mlsanity.com/paper/2108.10447/) (2021)
- [Fre-GAN: Adversarial Frequency-consistent Audio Synthesis](https://mlsanity.com/paper/2106.02297/) (2021)
- [GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech](https://mlsanity.com/paper/2205.07211/) (2022)
- [Phonological Features for 0-shot Multilingual Speech Synthesis](https://mlsanity.com/paper/2008.04107/) (2020)
- [Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features](https://mlsanity.com/paper/2203.03191/) (2022)
- [Heterogeneous Target Speech Separation](https://mlsanity.com/paper/2204.03594/) (2022)

## Built on by (most cited first, 3 of 3)

- [VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis](https://mlsanity.com/paper/2412.19259/) (2024)
- [CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation](https://mlsanity.com/paper/2412.20048/) (2024)
- [GSA-TTS : Toward Zero-Shot Speech Synthesis based on Gradual Style Adaptor](https://mlsanity.com/paper/2505.19384/) (2025)

## Abstract (arXiv)

While recent text-to-speech (TTS) systems have made remarkable strides toward human-level quality, the performance of cross-lingual TTS lags behind that of intra-lingual TTS. This gap is mainly rooted from the speaker-language entanglement problem in cross-lingual TTS. In this paper, we propose CrossSpeech which improves the quality of cross-lingual speech by effectively disentangling speaker and language information in the level of acoustic feature space. Specifically, CrossSpeech decomposes the speech generation pipeline into the speaker-independent generator (SIG) and speaker-dependent generator (SDG). The SIG produces the speaker-independent acoustic representation which is not biased to specific speaker distributions. On the other hand, the SDG models speaker-dependent speech variation that characterizes speaker attributes. By handling each information separately, CrossSpeech can obtain disentangled speaker and language representations. From the experiments, we verify that CrossSpeech achieves significant improvements in cross-lingual TTS, especially in terms of speaker similarity to the target speaker.