# SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

arXiv: https://arxiv.org/abs/2512.05126 · HTML: https://mlsanity.com/paper/2512.05126/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 23 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS](https://mlsanity.com/paper/2406.18009/) (2024)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision](https://mlsanity.com/paper/2412.09262/) (2024)
- [Neural Dubber: Dubbing for Videos According to Scripts](https://mlsanity.com/paper/2110.08243/) (2021)
- [ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching](https://mlsanity.com/paper/2506.13053/) (2025)
- [Learning to Dub Movies via Hierarchical Prosody Models](https://mlsanity.com/paper/2212.04054/) (2022)
- [StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing](https://mlsanity.com/paper/2402.12636/) (2024)
- [EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing](https://mlsanity.com/paper/2412.08988/) (2024)
- [VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over](https://mlsanity.com/paper/2110.03342/) (2021)
- [VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models](https://mlsanity.com/paper/2504.02386/) (2025)
- [Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing](https://mlsanity.com/paper/2503.12042/) (2025)
- [AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation](https://mlsanity.com/paper/2504.20629/) (2025)

## Abstract (arXiv)

Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.