# Multilingual Prosody Transfer: Comparing Supervised & Transfer Learning

arXiv: https://arxiv.org/abs/2406.00022 · HTML: https://mlsanity.com/paper/2406.00022/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 11 of 11)

- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining](https://mlsanity.com/paper/1912.06813/) (2019)
- [Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech](https://mlsanity.com/paper/2210.15447/) (2022)
- [A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation](https://mlsanity.com/paper/2301.10606/) (2023)
- [Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features](https://mlsanity.com/paper/1911.09645/) (2019)
- [Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfer](https://mlsanity.com/paper/2009.01475/) (2020)
- [A Vocoder-free WaveNet Voice Conversion with Non-Parallel Data](https://mlsanity.com/paper/1902.03705/) (2019)

## Built on by (most cited first, 2 of 2)

- [Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning](https://mlsanity.com/paper/2406.08931/) (2024)
- [MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation](https://mlsanity.com/paper/2604.18914/) (2026)

## Abstract (arXiv)

The field of prosody transfer in speech synthesis systems is rapidly advancing. This research is focused on evaluating learning methods for adapting pre-trained monolingual text-to-speech (TTS) models to multilingual conditions, i.e., Supervised Fine-Tuning (SFT) and Transfer Learning (TL). This comparison utilizes three distinct metrics: Mean Opinion Score (MOS), Recognition Accuracy (RA), and Mel Cepstral Distortion (MCD). Results demonstrate that, in comparison to SFT, TL leads to significantly enhanced performance, with an average MOS higher by 1.53 points, a 37.5% increase in RA, and approximately a 7.8-point improvement in MCD. These findings are instrumental in helping build TTS models for low-resource languages.