mlsanityData to 7 Oct 2026

V2C: Visual Voice Cloning

arXiv 2111.12890 ↗First posted 25 Nov 2021cs.CV cs.SD eess.AScited by 23

Not on the map. Why?

Abstract (arXiv)

Existing Voice Cloning (VC) tasks aim to convert a paragraph text to a speech with desired voice specified by a reference audio. This has significantly boosted the development of artificial speech applications. However, there also exist many scenarios that cannot be well reflected by these VC tasks, such as movie dubbing, which requires the speech to be with emotions consistent with the movie plots. To fill this gap, in this work we propose a new task named Visual Voice Cloning (V2C), which seeks to convert a paragraph of text to a speech with both desired voice specified by a reference audio and desired emotion specified by a reference video. To facilitate research in this field, we construct a dataset, V2C-Animation, and propose a strong baseline based on existing state-of-the-art (SoTA) VC techniques. Our dataset contains 10,217 animated movie clips covering a large variety of genres (e.g., Comedy, Fantasy) and emotions (e.g., happy, sad). We further design a set of evaluation metrics, named MCD-DTW-SL, which help evaluate the similarity between ground-truth speeches and the synthesised ones. Extensive experimental results show that even SoTA VC methods cannot generate satisfying speeches for our V2C task. We hope the proposed new task together with the constructed dataset and evaluation metric will facilitate the research in the field of voice cloning and the broader vision-and-language community.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 25 Nov 2021.

Builds on

36 of 55 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2016Least Squares Generative Adversarial Networkscited by 1,000
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  10. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  12. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
24 more
  1. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  2. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  3. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  4. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  5. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  6. 2016Lip Reading Sentences in the Wildcited by 226
  7. 2018Efficient Neural Audio Synthesiscited by 218
  8. 2015A Dataset for Movie Descriptioncited by 195
  9. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  10. 2018Neural Voice Cloning with a Few Samplescited by 100
  11. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  12. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  13. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  14. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  15. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  16. 2018Sample Efficient Adaptive Text-to-Speechcited by 45
  17. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  18. 2019Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokenscited by 44
  19. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 31
  20. 2018Fitting New Speakers Based on a Short Untranscribed Samplecited by 18
  21. 2021AdaSpeech 2: Adaptive Text to Speech with Untranscribed Datacited by 17
  22. 2019Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistencycited by 12
  23. 2019Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesiscited by 11
  24. 2019Data Efficient Voice Cloning for Neural Singing Synthesiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 5; 2025: 11; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  2. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  3. 2025Generative AI for Cel-Animation: A Surveycited by 17
  4. 2025Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbingcited by 9
  5. 2024Synchronous Multi-modal Semantic Communication System with Packet-level Codingcited by 7
  6. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  7. 2023Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversioncited by 3
  8. 2025Gaze-Guided Learning: Avoiding Shortcut Bias in Visual Classificationcited by 3
  9. 2025VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learningcited by 3
  10. 2026FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenescited by 3
  11. 2024Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesiscited by 2
  12. 2024Towards Expressive Video Dubbing with Multiscale Multimodal Context Interactioncited by 2
11 more
  1. 2026DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronizationcited by 2
  2. 2024VCoME: Verbal Video Composition with Multimodal Editing Effectscited by 1
  3. 2025Shushing! Let's Imagine an Authentic Speech from the Silent Videocited by 1
  4. 2025CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloningcited by 1
  5. 2025RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modelingcited by 1
  6. 2023Latent Optimal Paths by Gumbel Propagation for Variational Bayesian Dynamic Programmingcited by 0
  7. 2025MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translationcited by 0
  8. 2025Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarkscited by 0
  9. 2025MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbingcited by 0
  10. 2026CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbingcited by 0
  11. 2026Hierarchical Codec Diffusion for Video-to-Speech Generationcited by 0