mlsanityData to 7 Oct 2026

VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion

arXiv 2202.09081 ↗First posted 18 Feb 2022eess.AS cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Though significant progress has been made for speaker-dependent Video-to-Speech (VTS) synthesis, little attention is devoted to multi-speaker VTS that can map silent video to speech, while allowing flexible control of speaker identity, all in a single system. This paper proposes a novel multi-speaker VTS system based on cross-modal knowledge transfer from voice conversion (VC), where vector quantization with contrastive predictive coding (VQCPC) is used for the content encoder of VC to derive discrete phoneme-like acoustic units, which are transferred to a Lip-to-Index (Lip2Ind) network to infer the index sequence of acoustic units. The Lip2Ind network can then substitute the content encoder of VC to form a multi-speaker VTS system to convert silent video to acoustic units for reconstructing accurate spoken content. The VTS system also inherits the advantages of VC by using a speaker encoder to produce speaker representations to effectively control the speaker identity of generated speech. Extensive evaluations verify the effectiveness of proposed approach, which can be applied in both constrained vocabulary and open vocabulary conditions, achieving state-of-the-art performance in generating high-quality speech with high naturalness, intelligibility and speaker similarity. Our demo page is released here: https://wendison.github.io/VCVTS-demo/

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 18 Feb 2022.

Builds on

23 of 39 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  8. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  9. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  10. 2020Lipreading using Temporal Convolutional Networkscited by 98
  11. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  12. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
11 more
  1. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  2. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  3. 2017Lip2AudSpec: Speech reconstruction from silent lip movements videocited by 32
  4. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  5. 2017Improved Speech Reconstruction from Silent Videocited by 26
  6. 2019Video-Driven Speech Reconstruction using Generative Adversarial Networkscited by 25
  7. 2020Vocoder-Based Speech Synthesis from Silent Videoscited by 20
  8. 2019Lipper: Synthesizing Thy Speech using Multi-View Lipreadingcited by 19
  9. 2020Speech Prediction in Silent Videos using Variational Autoencoderscited by 18
  10. 2021Speaker disentanglement in video-to-speech conversioncited by 6
  11. 2018Harnessing AI for Speech Reconstruction using Multi-view Silent Video Feedcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 0; 2025: 1.

6 papers.

  1. 2023Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignmentcited by 8
  2. 2023LipLearner: Customizable Silent Speech Interactions on Mobile Devicescited by 5
  3. 2025NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processingcited by 3
  4. 2023Zero-shot personalized lip-to-speech synthesis with face image based voice controlcited by 2
  5. 2023Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networkscited by 1
  6. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1