# From Speech-to-Speech Translation to Automatic Dubbing

arXiv: https://arxiv.org/abs/2001.06785 · HTML: https://mlsanity.com/paper/2001.06785/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2020.

## Built on by (most cited first, 25 of 25)

- [AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages](https://mlsanity.com/paper/2109.04715/) (2021)
- [VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing](https://mlsanity.com/paper/2211.16934/) (2022)
- [Machine Translation Verbosity Control for Automatic Dubbing](https://mlsanity.com/paper/2110.03847/) (2021)
- [Isometric MT: Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08682/) (2021)
- [Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing](https://mlsanity.com/paper/2212.12137/) (2022)
- [Isochrony-Aware Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08548/) (2021)
- [Prosodic Alignment for off-screen automatic dubbing](https://mlsanity.com/paper/2204.02530/) (2022)
- [Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing](https://mlsanity.com/paper/2302.12979/) (2023)
- [Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning](https://mlsanity.com/paper/2304.05600/) (2023)
- [Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters](https://mlsanity.com/paper/2305.13204/) (2023)
- [Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization](https://mlsanity.com/paper/2508.08550/) (2025)
- [Technology Pipeline for Large Scale Cross-Lingual Dubbing of Lecture Videos into Multiple Indian Languages](https://mlsanity.com/paper/2211.01338/) (2022)
- [Prompting LLMs: Length Control for Isometric Machine Translation](https://mlsanity.com/paper/2506.04855/) (2025)
- [Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling](https://mlsanity.com/paper/2602.00597/) (2026)
- [IsoChronoMeter: A simple and effective isochronic translation evaluation metric](https://mlsanity.com/paper/2410.11127/) (2024)
- [Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing](https://mlsanity.com/paper/2505.20899/) (2025)
- [From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimization](https://mlsanity.com/paper/2602.01068/) (2026)
- [Evaluating the Efficacy of Length-Controllable Machine Translation](https://mlsanity.com/paper/2305.02300/) (2023)
- [Efficient Spoken Language Recognition via Multilabel Classification](https://mlsanity.com/paper/2306.01945/) (2023)
- [Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages](https://mlsanity.com/paper/2411.07387/) (2024)
- [MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation](https://mlsanity.com/paper/2503.11026/) (2025)
- [CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization](https://mlsanity.com/paper/2603.16966/) (2026)
- [PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing](https://mlsanity.com/paper/2604.09111/) (2026)
- [Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis](https://mlsanity.com/paper/2609.03992/) (2026)
- [Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing](https://mlsanity.com/paper/2610.00825/) (2026)

## Abstract (arXiv)

We present enhancements to a speech-to-speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of the translation with the original speech segments, neural text-to-speech with fine tuning of the duration of each utterance, and, finally, audio rendering to enriches text-to-speech output with background noise and reverberation extracted from the original audio. We report on a subjective evaluation of automatic dubbing of excerpts of TED Talks from English into Italian, which measures the perceived naturalness of automatic dubbing and the relative importance of each proposed enhancement.