mlsanityData to 7 Oct 2026

More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech

arXiv 2111.10139 ↗First posted 19 Nov 2021cs.CV cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper we present VDTTS, a Visually-Driven Text-to-Speech model. Motivated by dubbing, VDTTS takes advantage of video frames as an additional input alongside text, and generates speech that matches the video signal. We demonstrate how this allows VDTTS to, unlike plain TTS models, generate speech that not only has prosodic variations like natural pauses and pitch, but is also synchronized to the input video. Experimentally, we show our model produces well-synchronized outputs, approaching the video-speech synchronization quality of the ground-truth, on several challenging benchmarks including "in-the-wild" content from VoxCeleb2. Supplementary demo videos demonstrating video-speech synchronization, robustness to speaker ID swapping, and prosody, presented at the project page.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 19 Nov 2021.

Builds on

45 of 62 references.

References · 45

  1. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  10. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  11. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  12. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
33 more
  1. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  2. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  3. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  4. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  5. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  6. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  7. 2016Lip Reading Sentences in the Wildcited by 226
  8. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  9. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  10. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
  11. 2019Text-based Editing of Talking-head Videocited by 102
  12. 2020Towards Automatic Face-to-Face Translationcited by 95
  13. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  14. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  15. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  16. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  17. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  18. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  19. 2018Large-Scale Visual Speech Recognitioncited by 57
  20. 2017Vid2speech: Speech Reconstruction from Silent Videocited by 53
  21. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  22. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  23. 2019Neural Style-Preserving Visual Dubbingcited by 47
  24. 2017ObamaNet: Photo-realistic lip-sync from textcited by 42
  25. 2019Universal Sound Separationcited by 41
  26. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  27. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  28. 2021Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modelingcited by 31
  29. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  30. 2019Lipper: Synthesizing Thy Speech using Multi-View Lipreadingcited by 19
  31. 2020Large-scale multilingual audio visual dubbingcited by 15
  32. 2021VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Overcited by 13
  33. 2020Textual Echo Cancellationcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1; 2024: 4; 2025: 4.

12 papers.

  1. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  2. 2024StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbingcited by 20
  3. 2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8
  4. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  5. 2024V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flowcited by 5
  6. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
  7. 2023High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Unitscited by 4
  8. 2024Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesiscited by 2
  9. 2024Towards Expressive Video Dubbing with Multiscale Multimodal Context Interactioncited by 2
  10. 2022Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel's Weekly Video Podcastscited by 0
  11. 2025Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarkscited by 0
  12. 2025MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbingcited by 0