# MetaSpeech: Speech Effects Switch Along with Environment for Metaverse

arXiv: https://arxiv.org/abs/2210.13811 · HTML: https://mlsanity.com/paper/2210.13811/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda](https://mlsanity.com/paper/2110.05352/) (2021)
- [CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion](https://mlsanity.com/paper/1904.04631/) (2019)
- [Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder](https://mlsanity.com/paper/1610.04019/) (2016)
- [AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization](https://mlsanity.com/paper/2011.00316/) (2020)
- [FAST-RIR: Fast neural diffuse room impulse response generator](https://mlsanity.com/paper/2110.04057/) (2021)
- [Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1711.11293/) (2017)
- [CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversion](https://mlsanity.com/paper/2010.11672/) (2020)
- [AVQVC: One-shot Voice Conversion by Vector Quantization with applying contrastive learning](https://mlsanity.com/paper/2202.10020/) (2022)
- [MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames](https://mlsanity.com/paper/2102.12841/) (2021)
- [Metaverse for Social Good: A University Campus Prototype](https://mlsanity.com/paper/2108.08985/) (2021)
- [Realizing the Metaverse with Edge Intelligence: A Match Made in Heaven](https://mlsanity.com/paper/2201.01634/) (2022)
- [TS-RIR: Translated synthetic room impulse responses for speech augmentation](https://mlsanity.com/paper/2103.16804/) (2021)
- [Differentiable Signal Processing With Black-Box Audio Effects](https://mlsanity.com/paper/2105.04752/) (2021)
- [nnSpeech: Speaker-Guided Conditional Variational Autoencoder for Zero-shot Multi-speaker Text-to-Speech](https://mlsanity.com/paper/2202.10712/) (2022)
- [ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder](https://mlsanity.com/paper/1808.05092/) (2018)
- [DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning](https://mlsanity.com/paper/2202.10976/) (2022)
- [Non-autoregressive sequence-to-sequence voice conversion](https://mlsanity.com/paper/2104.06793/) (2021)
- [Global Rhythm Style Transfer Without Text Transcriptions](https://mlsanity.com/paper/2106.08519/) (2021)
- [Towards Low-Resource StarGAN Voice Conversion using Weight Adaptive Instance Normalization](https://mlsanity.com/paper/2010.11646/) (2020)
- [TGAVC: Improving Autoencoder Voice Conversion with Text-Guided and Adversarial Training](https://mlsanity.com/paper/2208.04035/) (2022)
- [Singer Identification Using Deep Timbre Feature Learning with KNN-Net](https://mlsanity.com/paper/2102.10236/) (2021)
- [Exploring Quality and Generalizability in Parameterized Neural Audio Effects](https://mlsanity.com/paper/2006.05584/) (2020)

## Abstract (arXiv)

Metaverse expands the physical world to a new dimension, and the physical environment and Metaverse environment can be directly connected and entered. Voice is an indispensable communication medium in the real world and Metaverse. Fusion of the voice with environment effects is important for user immersion in Metaverse. In this paper, we proposed using the voice conversion based method for the conversion of target environment effect speech. The proposed method was named MetaSpeech, which introduces an environment effect module containing an effect extractor to extract the environment information and an effect encoder to encode the environment effect condition, in which gradient reversal layer was used for adversarial training to keep the speech content and speaker information while disentangling the environmental effects. From the experiment results on the public dataset of LJSpeech with four environment effects, the proposed model could complete the specific environment effect conversion and outperforms the baseline methods from the voice conversion task.