# Cross-lingual Low Resource Speaker Adaptation Using Phonological Features

arXiv: https://arxiv.org/abs/2111.09075 · HTML: https://mlsanity.com/paper/2111.09075/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 17 Nov 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes](https://mlsanity.com/paper/1811.09021/) (2018)
- [One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech](https://mlsanity.com/paper/2008.00768/) (2020)
- [Phonological Features for 0-shot Multilingual Speech Synthesis](https://mlsanity.com/paper/2008.04107/) (2020)
- [Efficient neural speech synthesis for low-resource languages through multilingual modeling](https://mlsanity.com/paper/2008.09659/) (2020)
- [Cross-lingual Multispeaker Text-to-Speech under Limited-Data Scenario](https://mlsanity.com/paper/2005.10441/) (2020)
- [Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement](https://mlsanity.com/paper/2011.06392/) (2020)
- [Latent linguistic embedding for cross-lingual text-to-speech and voice conversion](https://mlsanity.com/paper/2010.03717/) (2020)

## Built on by (most cited first, 8 of 8)

- [Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)](https://mlsanity.com/paper/2207.01547/) (2022)
- [Mix and Match: An Empirical Study on Training Corpus Composition for Polyglot Text-To-Speech (TTS)](https://mlsanity.com/paper/2207.01507/) (2022)
- [Self-supervised learning for robust voice cloning](https://mlsanity.com/paper/2204.03421/) (2022)
- [DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarin](https://mlsanity.com/paper/2309.00883/) (2023)
- [Few-Shot Cross-Lingual TTS Using Transferable Phoneme Embedding](https://mlsanity.com/paper/2206.15427/) (2022)
- [Data-augmented cross-lingual synthesis in a teacher-student framework](https://mlsanity.com/paper/2204.00061/) (2022)
- [Integrating Human Linguistic Insights into AI: Theory-Driven Representation for Multilingual Text-to-Speech](https://mlsanity.com/paper/2204.07228/) (2022)
- [Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization](https://mlsanity.com/paper/2402.01692/) (2024)

## Abstract (arXiv)

The idea of using phonological features instead of phonemes as input to sequence-to-sequence TTS has been recently proposed for zero-shot multilingual speech synthesis. This approach is useful for code-switching, as it facilitates the seamless uttering of foreign text embedded in a stream of native text. In our work, we train a language-agnostic multispeaker model conditioned on a set of phonologically derived features common across different languages, with the goal of achieving cross-lingual speaker adaptation. We first experiment with the effect of language phonological similarity on cross-lingual TTS of several source-target language combinations. Subsequently, we fine-tune the model with very limited data of a new speaker's voice in either a seen or an unseen language, and achieve synthetic speech of equal quality, while preserving the target speaker's identity. With as few as 32 and 8 utterances of target speaker data, we obtain high speaker similarity scores and naturalness comparable to the corresponding literature. In the extreme case of only 2 available adaptation utterances, we find that our model behaves as a few-shot learner, as the performance is similar in both the seen and unseen adaptation language scenarios.