mlsanityData to 7 Oct 2026

OpenVoice: Versatile Instant Voice Cloning

arXiv 2312.01479 ↗First posted 3 Dec 2023cs.SD cs.LG eess.AScited by 30

Not on the map. Why?

Abstract (arXiv)

We introduce OpenVoice, a versatile voice cloning approach that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice represents a significant advancement in addressing the following open challenges in the field: 1) Flexible Voice Style Control. OpenVoice enables granular control over voice styles, including emotion, accent, rhythm, pauses, and intonation, in addition to replicating the tone color of the reference speaker. The voice styles are not directly copied from and constrained by the style of the reference speaker. Previous approaches lacked the ability to flexibly manipulate voice styles after cloning. 2) Zero-Shot Cross-Lingual Voice Cloning. OpenVoice achieves zero-shot cross-lingual voice cloning for languages not included in the massive-speaker training set. Unlike previous approaches, which typically require extensive massive-speaker multi-lingual (MSML) dataset for all languages, OpenVoice can clone voices into a new language without any massive-speaker training data for that language. OpenVoice is also computationally efficient, costing tens of times less than commercially available APIs that offer even inferior performance. To foster further research in the field, we have made the source code and trained model publicly accessible. We also provide qualitative results in our demo website. OpenVoice has been used by more than 2M users worldwide as the voice engine of MyShell.ai

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 3 Dec 2023.

Builds on

14 of 18 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  6. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  8. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  9. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  12. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
2 more
  1. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
  2. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 10; 2026: 8.

30 papers.

Sort:

By area:

  1. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  2. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  3. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  4. 2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
  5. 2025QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptionscited by 13
  6. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 7
  7. 2024Certification of Speaker Recognition Models to Additive Perturbationscited by 6
  8. 2024VoiceWukong: Benchmarking Deepfake Voice Detectioncited by 5
  9. 2025SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesiscited by 5
  10. 2024SecureSpectra: Safeguarding Digital Identity from Deep Fake Threats via Intelligent Signaturescited by 4
  11. 2024DiffSSD: A Diffusion-Based Dataset For Speech Forensicscited by 4
  12. 2025TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detectioncited by 4
18 more
  1. 2024The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspectivecited by 3
  2. 2025DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detectioncited by 3
  3. 2025De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attackscited by 2
  4. 2025A Survey of Threats Against Voice Authentication and Anti-Spoofing Systemscited by 2
  5. 2024Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systemscited by 1
  6. 2024SpMis: An Investigation of Synthetic Spoken Misinformation Detectioncited by 1
  7. 2024Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Modelcited by 1
  8. 2025StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversioncited by 1
  9. 2025Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systemscited by 1
  10. 2026ICo3D: An Interactive Conversational 3D Virtual Humancited by 1
  11. 2026An Evaluation Framework for Text-to-Speech Voice Reconstructioncited by 1
  12. 2025EchoFake: A Replay-Aware Dataset for Practical Speech Deepfake Detectioncited by 0
  13. 2026Talking Slide Avatars: Open-Source Multimodal Communication Approach for Teachingcited by 0
  14. 2026Elderly-Contextual Data Augmentation via Speech Synthesis for Elderly ASRcited by 0
  15. 2026V.O.I.C.E (Voice, Ownership, Identity, Control, Expression): Risk Taxonomy of Synthetic Voice Generation From Empirical Datacited by 0
  16. 2026UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videoscited by 0
  17. 2026Tracing Decoder Artifacts for Compact Synthetic Speech Screeningcited by 0
  18. 2026GLAD: Global-Local Adaptive Detector for Robust Speech Deepfake Detectioncited by 0