# SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention

arXiv: https://arxiv.org/abs/2312.08676 · HTML: https://mlsanity.com/paper/2312.08676/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 27 of 27)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization](https://mlsanity.com/paper/1904.05742/) (2019)
- [SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model](https://mlsanity.com/paper/2104.05557/) (2021)
- [Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder](https://mlsanity.com/paper/1610.04019/) (2016)
- [UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding](https://mlsanity.com/paper/2306.07547/) (2023)
- [VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature](https://mlsanity.com/paper/2204.00768/) (2022)
- [SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottlenecks](https://mlsanity.com/paper/2203.14156/) (2022)
- [Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversion](https://mlsanity.com/paper/2203.16705/) (2022)
- [Make-A-Voice: Unified Voice Synthesis With Discrete Representation](https://mlsanity.com/paper/2305.19269/) (2023)
- [Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations](https://mlsanity.com/paper/2010.12231/) (2020)
- [LM-VC: Zero-shot Voice Conversion via Speech Generation based on Language Models](https://mlsanity.com/paper/2306.10521/) (2023)
- [S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations](https://mlsanity.com/paper/2104.02901/) (2021)
- [ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations](https://mlsanity.com/paper/2302.08137/) (2023)
- [Towards Improved Zero-shot Voice Conversion with Conditional DSVAE](https://mlsanity.com/paper/2205.05227/) (2022)
- [DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning](https://mlsanity.com/paper/2202.10976/) (2022)
- [Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features](https://mlsanity.com/paper/2112.04424/) (2021)
- [DGC-vector: A new speaker embedding for zero-shot voice conversion](https://mlsanity.com/paper/2203.09722/) (2022)
- [NoiseVC: Towards High Quality Zero-Shot Voice Conversion](https://mlsanity.com/paper/2104.06074/) (2021)

## Built on by (most cited first, 19 of 19)

- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec](https://mlsanity.com/paper/2410.15764/) (2024)
- [vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders](https://mlsanity.com/paper/2409.01995/) (2024)
- [Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion](https://mlsanity.com/paper/2505.24291/) (2025)
- [Zero-Shot Voice Conversion via Content-Aware Timbre Ensemble and Conditional Flow Matching](https://mlsanity.com/paper/2411.02026/) (2024)
- [ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training](https://mlsanity.com/paper/2501.04416/) (2025)
- [VoicePrompter: Robust Zero-Shot Voice Conversion with Voice Prompt and Conditional Flow Matching](https://mlsanity.com/paper/2501.17612/) (2025)
- [EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion](https://mlsanity.com/paper/2505.16691/) (2025)
- [X-VC: Zero-shot Streaming Voice Conversion in Codec Space](https://mlsanity.com/paper/2604.12456/) (2026)
- [Discrete Unit based Masking for Improving Disentanglement in Voice Conversion](https://mlsanity.com/paper/2409.11560/) (2024)
- [Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling](https://mlsanity.com/paper/2410.01350/) (2024)
- [QR-VC: Leveraging Quantization Residuals for Linear Disentanglement in Zero-Shot Voice Conversion](https://mlsanity.com/paper/2411.16147/) (2024)
- [Noro: Noise-Robust One-shot Voice Conversion with Hidden Speaker Representation Learning](https://mlsanity.com/paper/2411.19770/) (2024)
- [Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching](https://mlsanity.com/paper/2506.01014/) (2025)
- [StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion](https://mlsanity.com/paper/2506.02414/) (2025)
- [Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosody](https://mlsanity.com/paper/2508.06890/) (2025)
- [CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion](https://mlsanity.com/paper/2411.18918/) (2024)
- [MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement](https://mlsanity.com/paper/2511.12074/) (2025)
- [The Voiceprint Fallacy: Why Voices Are Not Unique Biometric Imprints](https://mlsanity.com/paper/2608.07980/) (2026)

## Abstract (arXiv)

Zero-shot voice conversion (VC) aims to transfer the source speaker timbre to arbitrary unseen target speaker timbre, while keeping the linguistic content unchanged. Although the voice of generated speech can be controlled by providing the speaker embedding of the target speaker, the speaker similarity still lags behind the ground truth recordings. In this paper, we propose SEF-VC, a speaker embedding free voice conversion model, which is designed to learn and incorporate speaker timbre from reference speech via a powerful position-agnostic cross-attention mechanism, and then reconstruct waveform from HuBERT semantic tokens in a non-autoregressive manner. The concise design of SEF-VC enhances its training stability and voice conversion performance. Objective and subjective evaluations demonstrate the superiority of SEF-VC to generate high-quality speech with better similarity to target reference than strong zero-shot VC baselines, even for very short reference speeches.