# Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing

arXiv: https://arxiv.org/abs/2212.12137 · HTML: https://mlsanity.com/paper/2212.12137/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 74 references that are in the map.
- First posted on arXiv 23 Dec 2022.

## Builds on (most cited first, 17 of 17)

- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [VoxLingua107: a Dataset for Spoken Language Recognition](https://mlsanity.com/paper/2011.12998/) (2020)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Neural Style-Preserving Visual Dubbing](https://mlsanity.com/paper/1909.02518/) (2019)
- [Neural Dubber: Dubbing for Videos According to Scripts](https://mlsanity.com/paper/2110.08243/) (2021)
- [Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity](https://mlsanity.com/paper/2008.04935/) (2020)
- [From Speech-to-Speech Translation to Automatic Dubbing](https://mlsanity.com/paper/2001.06785/) (2020)
- [Machine Translation Verbosity Control for Automatic Dubbing](https://mlsanity.com/paper/2110.03847/) (2021)
- [Isometric MT: Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08682/) (2021)
- [Isochrony-Aware Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08548/) (2021)
- [Prosodic Phrase Alignment for Machine Dubbing](https://mlsanity.com/paper/1908.07226/) (2019)
- [Exploiting Sentence Order in Document Alignment](https://mlsanity.com/paper/2004.14523/) (2020)
- [Prosodic Alignment for off-screen automatic dubbing](https://mlsanity.com/paper/2204.02530/) (2022)

## Built on by (most cited first, 17 of 17)

- [A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism](https://mlsanity.com/paper/2401.05749/) (2024)
- [ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video](https://mlsanity.com/paper/2401.05314/) (2024)
- [Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing](https://mlsanity.com/paper/2302.12979/) (2023)
- [Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters](https://mlsanity.com/paper/2305.13204/) (2023)
- [End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation](https://mlsanity.com/paper/2311.00697/) (2023)
- [Speaker Diarization of Scripted Audiovisual Content](https://mlsanity.com/paper/2308.02160/) (2023)
- [Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems](https://mlsanity.com/paper/2407.13153/) (2024)
- [IsoChronoMeter: A simple and effective isochronic translation evaluation metric](https://mlsanity.com/paper/2410.11127/) (2024)
- [Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages](https://mlsanity.com/paper/2608.27848/) (2026)
- [LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformation](https://mlsanity.com/paper/2409.09326/) (2024)
- [Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation](https://mlsanity.com/paper/2412.16530/) (2024)
- [MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation](https://mlsanity.com/paper/2503.11026/) (2025)
- [Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?](https://mlsanity.com/paper/2603.19831/) (2026)
- [PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing](https://mlsanity.com/paper/2604.09111/) (2026)
- [Benchmarking Speech-to-Speech Translation Models](https://mlsanity.com/paper/2606.03241/) (2026)
- [Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing](https://mlsanity.com/paper/2609.26486/) (2026)
- [Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing](https://mlsanity.com/paper/2610.00825/) (2026)

## Abstract (arXiv)

We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as semantic transfer such as emphasis/emotion, in automatic dubbing systems.