mlsanityData to 7 Oct 2026

VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over

arXiv 2110.03342 ↗First posted 7 Oct 2021eess.AS cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we formulate a novel task to synthesize speech in sync with a silent pre-recorded video, denoted as automatic voice over (AVO). Unlike traditional speech synthesis, AVO seeks to generate not only human-sounding speech, but also perfect lip-speech synchronization. A natural solution to AVO is to condition the speech rendering on the temporal progression of lip sequence in the video. We propose a novel text-to-speech model that is conditioned on visual input, named VisualTTS, for accurate lip-speech synchronization. The proposed VisualTTS adopts two novel mechanisms that are 1) textual-visual attention, and 2) visual fusion strategy during acoustic decoding, which both contribute to forming accurate alignment between the input text content and lip motion in input lip sequence. Experimental results show that VisualTTS achieves accurate lip-speech synchronization and outperforms all baseline systems.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

23 of 33 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  7. 2018Deep Audio-Visual Speech Recognitioncited by 318
  8. 2018Efficient Neural Audio Synthesiscited by 218
  9. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
  10. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
  11. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  12. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 79
11 more
  1. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
  2. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  3. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  4. 2019Time Domain Audio Visual Speech Separationcited by 30
  5. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  6. 2020Multi-modal Attention for Speech Emotion Recognitioncited by 29
  7. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  8. 2020Muse: Multi-modal target speaker extraction with visual cuescited by 21
  9. 2018Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent languagecited by 14
  10. 2020Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancementcited by 5
  11. 2018A comparison of recent waveform generation and acoustic modeling methods for neural-network-based speech synthesiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 2; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  2. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  3. 2024StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbingcited by 20
  4. 2021More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speechcited by 12
  5. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 10
  6. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  7. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  8. 2023High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Unitscited by 4
  9. 2025FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Stylescited by 4
  10. 2022Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel's Weekly Video Podcastscited by 0
  11. 2025SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTScited by 0
  12. 2025Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systemscited by 0
1 more
  1. 2026Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?cited by 0