mlsanityData to 7 Oct 2026

SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention

arXiv 2312.08676 ↗First posted 14 Dec 2023cs.SD cs.CL eess.AScited by 19

Not on the map. Why?

Abstract (arXiv)

Zero-shot voice conversion (VC) aims to transfer the source speaker timbre to arbitrary unseen target speaker timbre, while keeping the linguistic content unchanged. Although the voice of generated speech can be controlled by providing the speaker embedding of the target speaker, the speaker similarity still lags behind the ground truth recordings. In this paper, we propose SEF-VC, a speaker embedding free voice conversion model, which is designed to learn and incorporate speaker timbre from reference speech via a powerful position-agnostic cross-attention mechanism, and then reconstruct waveform from HuBERT semantic tokens in a non-autoregressive manner. The concise design of SEF-VC enhances its training stability and voice conversion performance. Objective and subjective evaluations demonstrate the superiority of SEF-VC to generate high-quality speech with better similarity to target reference than strong zero-shot VC baselines, even for very short reference speeches.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

27 of 33 references.

References · 27

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  7. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  8. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
  11. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  12. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
15 more
  1. 2016Voice Conversion from Non-parallel Corpora Using Variational Auto-encodercited by 45
  2. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  3. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  4. 2022Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversioncited by 28
  5. 2022SpeechSplit 2.0: Unsupervised speech disentanglement for voice conversion Without tuning autoencoder Bottleneckscited by 27
  6. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  7. 2020Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representationscited by 18
  8. 2023LM-VC: Zero-shot Voice Conversion via Speech Generation based on Language Modelscited by 18
  9. 2021S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representationscited by 17
  10. 2023ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representationscited by 15
  11. 2022Towards Improved Zero-shot Voice Conversion with Conditional DSVAEcited by 10
  12. 2022DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learningcited by 9
  13. 2021Training Robust Zero-Shot Voice Conversion Models with Self-supervised Featurescited by 7
  14. 2022DGC-vector: A new speaker embedding for zero-shot voice conversioncited by 5
  15. 2021NoiseVC: Towards High Quality Zero-Shot Voice Conversioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 2.

19 papers.

Sort:

By area:

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
  3. 2024vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoderscited by 6
  4. 2025Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversioncited by 5
  5. 2024Zero-Shot Voice Conversion via Content-Aware Timbre Ensemble and Conditional Flow Matchingcited by 3
  6. 2025ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Trainingcited by 3
  7. 2025VoicePrompter: Robust Zero-Shot Voice Conversion with Voice Prompt and Conditional Flow Matchingcited by 3
  8. 2025EZ-VC: Easy Zero-shot Any-to-Any Voice Conversioncited by 2
  9. 2026X-VC: Zero-shot Streaming Voice Conversion in Codec Spacecited by 2
  10. 2024Discrete Unit based Masking for Improving Disentanglement in Voice Conversioncited by 1
  11. 2024Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modelingcited by 1
  12. 2024QR-VC: Leveraging Quantization Residuals for Linear Disentanglement in Zero-Shot Voice Conversioncited by 1
7 more
  1. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matchingcited by 1
  2. 2025StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversioncited by 1
  3. 2025Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosodycited by 1
  4. 2024CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversioncited by 0
  5. 2024Noro: Noise-Robust One-shot Voice Conversion with Hidden Speaker Representation Learningcited by 0
  6. 2025MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglementcited by 0
  7. 2026The Voiceprint Fallacy: Why Voices Are Not Unique Biometric Imprintscited by 0