mlsanityData to 7 Oct 2026

Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing

arXiv 2302.12979 ↗First posted 25 Feb 2023cs.CL cs.SD eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Automatic dubbing (AD) is the task of translating the original speech in a video into target language speech. The new target language speech should satisfy isochrony; that is, the new speech should be time aligned with the original video, including mouth movements, pauses, hand gestures, etc. In this paper, we propose training a model that directly optimizes both the translation as well as the speech duration of the generated translations. We show that this system generates speech that better matches the timing of the original speech, compared to prior work, while simplifying the system architecture.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 25 Feb 2023.

Builds on

16 of 27 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  7. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  8. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  9. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  10. 2020From Speech-to-Speech Translation to Automatic Dubbingcited by 25
  11. 2022VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbingcited by 20
  12. 2021Machine Translation Verbosity Control for Automatic Dubbingcited by 18
4 more
  1. 2021Isometric MT: Neural Machine Translation for Automatic Dubbingcited by 18
  2. 2022Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbingcited by 17
  3. 2021Isochrony-Aware Neural Machine Translation for Automatic Dubbingcited by 14
  4. 2019Prosodic Phrase Alignment for Machine Dubbingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 0; 2026: 1.

5 papers.

  1. 2023Improving Isochronous Machine Translation with Target Factors and Auxiliary Counterscited by 3
  2. 2024IsoChronoMeter: A simple and effective isochronic translation evaluation metriccited by 1
  3. 2024Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languagescited by 0
  4. 2024Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translationcited by 0
  5. 2026Benchmarking Speech-to-Speech Translation Modelscited by 0