# MoEVC: A Mixture-of-experts Voice Conversion System with Sparse Gating Mechanism for Accelerating Online Computation

arXiv: https://arxiv.org/abs/1912.11984 · HTML: https://mlsanity.com/paper/1912.11984/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 27 Dec 2019.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Data-Driven Sparse Structure Selection for Deep Neural Networks](https://mlsanity.com/paper/1707.01213/) (2017)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder](https://mlsanity.com/paper/1610.04019/) (2016)
- [Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks](https://mlsanity.com/paper/1704.00849/) (2017)
- [Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations](https://mlsanity.com/paper/1804.02812/) (2018)
- [AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanisms](https://mlsanity.com/paper/1811.04076/) (2018)
- [ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder](https://mlsanity.com/paper/1808.05092/) (2018)
- [Voice Conversion Based on Cross-Domain Features Using Variational Auto Encoders](https://mlsanity.com/paper/1808.09634/) (2018)
- [Investigation of F0 conditioning and Fully Convolutional Networks in Variational Autoencoder based Voice Conversion](https://mlsanity.com/paper/1905.00615/) (2019)
- [Precision Scaling of Neural Networks for Efficient Audio Processing](https://mlsanity.com/paper/1712.01340/) (2017)
- [A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN)](https://mlsanity.com/paper/1808.06474/) (2018)
- [A Vocoder-free WaveNet Voice Conversion with Non-Parallel Data](https://mlsanity.com/paper/1902.03705/) (2019)

## Abstract (arXiv)

With the recent advancements of deep learning technologies, the performance of voice conversion (VC) in terms of quality and similarity has been significantly improved. However, heavy computations are generally required for deep-learning-based VC systems, which can cause notable latency and thus confine their deployments in real-world applications. Therefore, increasing online computation efficiency has become an important task. In this study, we propose a novel mixture-of-experts (MoE) based VC system. The MoE model uses a gating mechanism to specify optimal weights to feature maps to increase VC performance. In addition, assigning sparse constraints on the gating mechanism can accelerate online computation by skipping the convolution process by zeroing out redundant feature maps. Experimental results show that by specifying suitable sparse constraints, we can effectively increase the online computation efficiency with a notable 70% FLOPs (floating-point operations per second) reduction while improving the VC performance in both objective evaluations and human listening tests.