# Isochrony-Aware Neural Machine Translation for Automatic Dubbing

arXiv: https://arxiv.org/abs/2112.08548 · HTML: https://mlsanity.com/paper/2112.08548/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Positional Encoding to Control Output Sequence Length](https://mlsanity.com/paper/1904.07418/) (2019)
- [Linguistic Input Features Improve Neural Machine Translation](https://mlsanity.com/paper/1606.02892/) (2016)
- [Controlling the Output Length of Neural Machine Translation](https://mlsanity.com/paper/1910.10408/) (2019)
- [From Speech-to-Speech Translation to Automatic Dubbing](https://mlsanity.com/paper/2001.06785/) (2020)
- [Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?](https://mlsanity.com/paper/2006.01080/) (2020)
- [Machine Translation Verbosity Control for Automatic Dubbing](https://mlsanity.com/paper/2110.03847/) (2021)
- [MuST-Cinema: a Speech-to-Subtitles corpus](https://mlsanity.com/paper/2002.10829/) (2020)
- [Predicting Target Language CCG Supertags Improves Neural Machine Translation](https://mlsanity.com/paper/1702.01147/) (2017)
- [Prosodic Phrase Alignment for Machine Dubbing](https://mlsanity.com/paper/1908.07226/) (2019)
- [Modeling Target-Side Inflection in Neural Machine Translation](https://mlsanity.com/paper/1707.06012/) (2017)
- [Machine Translation with Unsupervised Length-Constraints](https://mlsanity.com/paper/2004.03176/) (2020)
- [Novel Applications of Factored Neural Machine Translation](https://mlsanity.com/paper/1910.03912/) (2019)
- [Factored Neural Machine Translation](https://mlsanity.com/paper/1609.04621/) (2016)
- [Joint translation and unit conversion for end-to-end localization](https://mlsanity.com/paper/2004.05219/) (2020)

## Built on by (most cited first, 14 of 14)

- [VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing](https://mlsanity.com/paper/2211.16934/) (2022)
- [Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing](https://mlsanity.com/paper/2212.12137/) (2022)
- [Direct Speech Translation for Automatic Subtitling](https://mlsanity.com/paper/2209.13192/) (2022)
- [Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023](https://mlsanity.com/paper/2309.15554/) (2023)
- [ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video](https://mlsanity.com/paper/2401.05314/) (2024)
- [Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing](https://mlsanity.com/paper/2302.12979/) (2023)
- [Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters](https://mlsanity.com/paper/2305.13204/) (2023)
- [Prompting LLMs: Length Control for Isometric Machine Translation](https://mlsanity.com/paper/2506.04855/) (2025)
- [SBAAM! Eliminating Transcript Dependency in Automatic Subtitling](https://mlsanity.com/paper/2405.10741/) (2024)
- [Length Aware Speech Translation for Video Dubbing](https://mlsanity.com/paper/2506.00740/) (2025)
- [Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages](https://mlsanity.com/paper/2411.07387/) (2024)
- [PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing](https://mlsanity.com/paper/2604.09111/) (2026)
- [Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing](https://mlsanity.com/paper/2609.26486/) (2026)
- [Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing](https://mlsanity.com/paper/2610.00825/) (2026)

## Abstract (arXiv)

We introduce the task of isochrony-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the speech-pause structure of the source into the target language to achieve audiovisual coherence. Practically, this implies correctly projecting pauses from the source to the target and ensuring that target speech segments have roughly the same duration of the corresponding source speech segments. In this work, we propose implicit and explicit modeling approaches to integrate isochrony information into neural machine translation. Experiments on English-German/French language pairs with automatic metrics show that the simplest of the considered approaches works best. Results are confirmed by human evaluations of translations and dubbed videos.