# Pureformer-VC: Non-parallel One-Shot Voice Conversion with Pure Transformer Blocks and Triplet Discriminative Training

arXiv: https://arxiv.org/abs/2409.01668 · HTML: https://mlsanity.com/paper/2409.01668/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks](https://mlsanity.com/paper/1806.02169/) (2018)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion](https://mlsanity.com/paper/2107.10394/) (2021)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversion](https://mlsanity.com/paper/1907.12279/) (2019)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversion](https://mlsanity.com/paper/2010.11672/) (2020)
- [A Batch Normalized Inference Network Keeps the KL Vanishing Away](https://mlsanity.com/paper/2004.12585/) (2020)
- [Towards Low-Resource StarGAN Voice Conversion using Weight Adaptive Instance Normalization](https://mlsanity.com/paper/2010.11646/) (2020)
- [MAIN-VC: Lightweight Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2405.00930/) (2024)

## Abstract (arXiv)

One-shot voice conversion(VC) aims to change the timbre of any source speech to match that of the target speaker with only one speech sample. Existing style transfer-based VC methods relied on speech representation disentanglement and suffered from accurately and independently encoding each speech component and recomposing back to converted speech effectively. To tackle this, we proposed Pureformer-VC, which utilizes Conformer blocks to build a disentangled encoder, and Zipformer blocks to build a style transfer decoder as the generator. In the decoder, we used effective styleformer blocks to integrate speaker characteristics effectively into the generated speech. The models used the generative VAE loss for encoding components and triplet loss for unsupervised discriminative training. We applied the styleformer method to Zipformer's shared weights for style transfer. The experimental results show that the proposed model achieves comparable subjective scores and exhibits improvements in objective metrics compared to existing methods in a one-shot voice conversion scenario.