# Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques

arXiv: https://arxiv.org/abs/2104.00931 · HTML: https://mlsanity.com/paper/2104.00931/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 27 of 27)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens](https://mlsanity.com/paper/1910.11997/) (2019)
- [wav2letter++: The Fastest Open-source Speech Recognition System](https://mlsanity.com/paper/1812.07625/) (2018)
- [F0-consistent many-to-many non-parallel voice conversion via conditional autoencoder](https://mlsanity.com/paper/2004.07370/) (2020)
- [Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion](https://mlsanity.com/paper/1906.00794/) (2019)
- [Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining](https://mlsanity.com/paper/1912.06813/) (2019)
- [Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet](https://mlsanity.com/paper/1903.12389/) (2019)
- [TTS Skins: Speaker Conversion via ASR](https://mlsanity.com/paper/1904.08983/) (2019)
- [Cotatron: Transcription-Guided Speech Encoder for Any-to-Many Voice Conversion without Parallel Data](https://mlsanity.com/paper/2005.03295/) (2020)
- [Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision](https://mlsanity.com/paper/1811.08111/) (2018)
- [Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfer](https://mlsanity.com/paper/2009.01475/) (2020)
- [Towards Fine-Grained Prosody Control for Voice Conversion](https://mlsanity.com/paper/1910.11269/) (2019)
- [Transferring Source Style in Non-Parallel Voice Conversion](https://mlsanity.com/paper/2005.09178/) (2020)
- [Rhythm-Flexible Voice Conversion without Parallel Data Using Cycle-GAN over Phoneme Posteriorgram Sequences](https://mlsanity.com/paper/1808.03113/) (2018)
- [Phonetic Posteriorgrams based Many-to-Many Singing Voice Conversion via Adversarial Training](https://mlsanity.com/paper/2012.01837/) (2020)
- [Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech](https://mlsanity.com/paper/1909.06532/) (2019)

## Built on by (most cited first, 11 of 11)

- [Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme](https://mlsanity.com/paper/2109.13821/) (2021)
- [Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units](https://mlsanity.com/paper/2212.09730/) (2022)
- [JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAE](https://mlsanity.com/paper/2210.06007/) (2022)
- [Deepfake CAPTCHA: A Method for Preventing Fake Calls](https://mlsanity.com/paper/2301.03064/) (2023)
- [PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTS](https://mlsanity.com/paper/2302.12391/) (2023)
- [HiFi-VC: High Quality ASR-Based Voice Conversion](https://mlsanity.com/paper/2203.16937/) (2022)
- [Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversion](https://mlsanity.com/paper/2311.08104/) (2023)
- [DisC-VC: Disentangled and F0-Controllable Neural Voice Conversion](https://mlsanity.com/paper/2210.11059/) (2022)
- [Artificial ASMR: A Cyber-Psychological Approach](https://mlsanity.com/paper/2210.14321/) (2022)
- [Linguistically Augmented Audio Speech Data (LinguAS)](https://mlsanity.com/paper/2606.10246/) (2026)
- [Deep-Fake CAPTCHA: Mitigating Next-Generation Social Engineering Attacks](https://mlsanity.com/paper/2609.11404/) (2026)

## Abstract (arXiv)

Recent works on voice conversion (VC) focus on preserving the rhythm and the intonation as well as the linguistic content. To preserve these features from the source, we decompose current non-parallel VC systems into two encoders and one decoder. We analyze each module with several experiments and reassemble the best components to propose Assem-VC, a new state-of-the-art any-to-many non-parallel VC system. We also examine that PPG and Cotatron features are speaker-dependent, and attempt to remove speaker identity with adversarial training. Code and audio samples are available at https://github.com/mindslab-ai/assem-vc.