mlsanityData to 7 Oct 2026

Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing

arXiv 2212.12137 ↗First posted 23 Dec 2022cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as semantic transfer such as emphasis/emotion, in automatic dubbing systems.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 17 of its 74 references that are in the map. First posted on arXiv 23 Dec 2022.

Builds on

17 of 74 references.

References · 17

  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  3. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 175
  4. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  5. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  6. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  7. 2021End-to-end speaker segmentation for overlap-aware resegmentationcited by 76
  8. 2019Neural Style-Preserving Visual Dubbingcited by 47
  9. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  10. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  11. 2020From Speech-to-Speech Translation to Automatic Dubbingcited by 25
  12. 2021Machine Translation Verbosity Control for Automatic Dubbingcited by 18
5 more
  1. 2021Isometric MT: Neural Machine Translation for Automatic Dubbingcited by 18
  2. 2021Isochrony-Aware Neural Machine Translation for Automatic Dubbingcited by 14
  3. 2019Prosodic Phrase Alignment for Machine Dubbingcited by 13
  4. 2020Exploiting Sentence Order in Document Alignmentcited by 13
  5. 2022Prosodic Alignment for off-screen automatic dubbingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 1; 2026: 6.

17 papers.

Sort:

By area:

  1. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  2. 2024ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Videocited by 6
  3. 2023Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbingcited by 5
  4. 2023Improving Isochronous Machine Translation with Target Factors and Auxiliary Counterscited by 3
  5. 2023End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translationcited by 2
  6. 2023Speaker Diarization of Scripted Audiovisual Contentcited by 1
  7. 2024Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systemscited by 1
  8. 2024IsoChronoMeter: A simple and effective isochronic translation evaluation metriccited by 1
  9. 2026Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languagescited by 1
  10. 2024LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformationcited by 0
  11. 2024Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translationcited by 0
  12. 2025MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translationcited by 0
5 more
  1. 2026Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?cited by 0
  2. 2026PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbingcited by 0
  3. 2026Benchmarking Speech-to-Speech Translation Modelscited by 0
  4. 2026Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbingcited by 0
  5. 2026Align Then Reason: A Multimodal Lip-Sync Judge for Dubbingcited by 0