# OpenVoice: Versatile Instant Voice Cloning

arXiv: https://arxiv.org/abs/2312.01479 · HTML: https://mlsanity.com/paper/2312.01479/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 3 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)

## Built on by (most cited first, 30 of 32)

- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [Zero-shot Voice Conversion with Diffusion Transformers](https://mlsanity.com/paper/2411.09943/) (2024)
- [SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters](https://mlsanity.com/paper/2412.00174/) (2024)
- [TTSDS -- Text-to-Speech Distribution Score](https://mlsanity.com/paper/2407.12707/) (2024)
- [QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions](https://mlsanity.com/paper/2503.20290/) (2025)
- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [Certification of Speaker Recognition Models to Additive Perturbations](https://mlsanity.com/paper/2404.18791/) (2024)
- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [VoiceWukong: Benchmarking Deepfake Voice Detection](https://mlsanity.com/paper/2409.06348/) (2024)
- [SecureSpectra: Safeguarding Digital Identity from Deep Fake Threats via Intelligent Signatures](https://mlsanity.com/paper/2407.00913/) (2024)
- [DiffSSD: A Diffusion-Based Dataset For Speech Forensics](https://mlsanity.com/paper/2409.13049/) (2024)
- [TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection](https://mlsanity.com/paper/2512.10652/) (2025)
- [The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective](https://mlsanity.com/paper/2405.08720/) (2024)
- [DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection](https://mlsanity.com/paper/2510.22622/) (2025)
- [De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks](https://mlsanity.com/paper/2507.02606/) (2025)
- [A Survey of Threats Against Voice Authentication and Anti-Spoofing Systems](https://mlsanity.com/paper/2508.16843/) (2025)
- [Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems](https://mlsanity.com/paper/2407.13153/) (2024)
- [SpMis: An Investigation of Synthetic Spoken Misinformation Detection](https://mlsanity.com/paper/2409.11308/) (2024)
- [Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model](https://mlsanity.com/paper/2411.05903/) (2024)
- [StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion](https://mlsanity.com/paper/2506.02414/) (2025)
- [Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems](https://mlsanity.com/paper/2509.07677/) (2025)
- [ICo3D: An Interactive Conversational 3D Virtual Human](https://mlsanity.com/paper/2601.13148/) (2026)
- [An Evaluation Framework for Text-to-Speech Voice Reconstruction](https://mlsanity.com/paper/2606.21343/) (2026)
- [EchoFake: A Replay-Aware Dataset for Practical Speech Deepfake Detection](https://mlsanity.com/paper/2510.19414/) (2025)
- [Talking Slide Avatars: Open-Source Multimodal Communication Approach for Teaching](https://mlsanity.com/paper/2604.23703/) (2026)
- [Elderly-Contextual Data Augmentation via Speech Synthesis for Elderly ASR](https://mlsanity.com/paper/2604.24770/) (2026)
- [V.O.I.C.E (Voice, Ownership, Identity, Control, Expression): Risk Taxonomy of Synthetic Voice Generation From Empirical Data](https://mlsanity.com/paper/2604.24794/) (2026)
- [UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos](https://mlsanity.com/paper/2608.11752/) (2026)
- [Tracing Decoder Artifacts for Compact Synthetic Speech Screening](https://mlsanity.com/paper/2609.32050/) (2026)
- [GLAD: Global-Local Adaptive Detector for Robust Speech Deepfake Detection](https://mlsanity.com/paper/2609.35411/) (2026)

## Abstract (arXiv)

We introduce OpenVoice, a versatile voice cloning approach that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice represents a significant advancement in addressing the following open challenges in the field: 1) Flexible Voice Style Control. OpenVoice enables granular control over voice styles, including emotion, accent, rhythm, pauses, and intonation, in addition to replicating the tone color of the reference speaker. The voice styles are not directly copied from and constrained by the style of the reference speaker. Previous approaches lacked the ability to flexibly manipulate voice styles after cloning. 2) Zero-Shot Cross-Lingual Voice Cloning. OpenVoice achieves zero-shot cross-lingual voice cloning for languages not included in the massive-speaker training set. Unlike previous approaches, which typically require extensive massive-speaker multi-lingual (MSML) dataset for all languages, OpenVoice can clone voices into a new language without any massive-speaker training data for that language. OpenVoice is also computationally efficient, costing tens of times less than commercially available APIs that offer even inferior performance. To foster further research in the field, we have made the source code and trained model publicly accessible. We also provide qualitative results in our demo website. OpenVoice has been used by more than 2M users worldwide as the voice engine of MyShell.ai