MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Not on the map. Why?
Abstract (arXiv)
Voice conversion aims to modify the source speaker's voice to resemble the target speaker while preserving the original speech content. Despite notable advancements in voice conversion these days, multi-lingual voice conversion (including both monolingual and cross-lingual scenarios) has yet to be extensively studied. It faces two main challenges: 1) the considerable variability in prosody and articulation habits across languages; and 2) the rarity of paired multi-lingual datasets from the same speaker. In this paper, we propose MulliVC, a novel voice conversion system that only converts timbre and keeps original content and source language prosody without multi-lingual paired data. Specifically, each training step of MulliVC contains three substeps: In step one the model is trained with monolingual speech data; then, steps two and three take inspiration from back translation, construct a cyclical process to disentangle the timbre and other information (content, prosody, and other language-related information) in the absence of multi-lingual data from the same speaker. Both objective and subjective results indicate that MulliVC significantly surpasses other methods in both monolingual and cross-lingual contexts, demonstrating the system's efficacy and the viability of the three-step approach with cycle consistency. Audio samples can be found on our demo page (mullivc.github.io).
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 8 Aug 2024.
Builds on
25 of 50 references.
References · 25
- 2017Attention Is All You Needcited by 59,932
- 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2016Least Squares Generative Adversarial Networkscited by 1,000
- 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
13 more
- 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
- 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
- 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
- 2016Dual Learning for Machine Translationcited by 146
- 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
- 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
- 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
- 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 71
- 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
- 2022Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Trainingcited by 7
- 2023Using joint training speaker encoder with consistency loss to achieve cross-lingual voice conversion and expressive voice conversioncited by 5
- 2021Towards Natural and Controllable Cross-Lingual Voice Conversion Based on Neural TTS Model and Phonetic Posteriorgramcited by 3
- 2022An Empirical Study on L2 Accents of Cross-lingual Text-to-Speech Systems via Vowel Spacecited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.