mlsanityData to 7 Oct 2026

A Comparative Study of Self-supervised Speech Representation Based Voice Conversion

arXiv 2207.04356 ↗First posted 10 Jul 2022cs.SD cs.LG eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

We present a large-scale comparative study of self-supervised speech representation (S3R)-based voice conversion (VC). In the context of recognition-synthesis VC, S3Rs are attractive owing to their potential to replace expensive supervised representations such as phonetic posteriorgrams (PPGs), which are commonly adopted by state-of-the-art VC systems. Using S3PRL-VC, an open-source VC software we previously developed, we provide a series of in-depth objective and subjective analyses under three VC settings: intra-/cross-lingual any-to-one (A2O) and any-to-any (A2A) VC, using the voice conversion challenge 2020 (VCC2020) dataset. We investigated S3R-based VC in various aspects, including model type, multilinguality, and supervision. We also studied the effect of a post-discretization process with k-means clustering and showed how it improves in the A2A setting. Finally, the comparison with state-of-the-art VC systems demonstrates the competitiveness of S3R-based VC and also sheds light on the possible improving directions.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 45 of its 78 references that are in the map. First posted on arXiv 10 Jul 2022.

Builds on

45 of 78 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  10. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
33 more
  1. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  2. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  3. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  4. 2021Unsupervised Speech Recognitioncited by 147
  5. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  6. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
  7. 2020Unsupervised pretraining transfers well across languagescited by 100
  8. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  9. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  10. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  11. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  12. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  13. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  14. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  15. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  16. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  17. 2016Voice Conversion from Non-parallel Corpora Using Variational Auto-encodercited by 45
  18. 2017Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networkscited by 38
  19. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  20. 2020An Open source Implementation of ITU-T Recommendation P.808 with Validationcited by 27
  21. 2019VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019cited by 25
  22. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  23. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 22
  24. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
  25. 2020Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissionscited by 19
  26. 2020Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representationscited by 18
  27. 2021S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representationscited by 17
  28. 2020The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTScited by 12
  29. 2020Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfercited by 10
  30. 2020Transferring Source Style in Non-Parallel Voice Conversioncited by 7
  31. 2021On Prosody Modeling for ASR+TTS based Voice Conversioncited by 3
  32. 2018Refined WaveNet Vocoder for Variational Autoencoder Based Voice Conversioncited by 2
  33. 2020The Academia Sinica Systems of Voice Conversion for VCC2020cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 3; 2025: 1; 2026: 1.

9 papers.

  1. 2023The Singing Voice Conversion Challenge 2023cited by 25
  2. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  3. 2024Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representationscited by 7
  4. 2023Evaluating Methods for Ground-Truth-Free Foreign Accent Conversioncited by 4
  5. 2025Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equationcited by 3
  6. 2023A Comparative Study of Voice Conversion Models with Large-Scale Speech and Singing Data: The T13 Systems for the Singing Voice Conversion Challenge 2023cited by 2
  7. 2024SingVisio: Visual Analytics of Diffusion Model for Singing Voice Conversioncited by 2
  8. 2024A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowingscited by 1
  9. 2026NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymizationcited by 0