mlsanityData to 14 Oct 2026

Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters

arXiv 2305.13204 ↗First posted 22 May 2023cs.CL cs.SD eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

To translate speech for automatic dubbing, machine translation needs to be isochronous, i.e. translated speech needs to be aligned with the source in terms of speech durations. We introduce target factors in a transformer model to predict durations jointly with target language phoneme sequences. We also introduce auxiliary counters to help the decoder to keep track of the timing information while generating target phonemes. We show that our model improves translation quality and isochrony compared to previous work where the translation model is instead trained to predict interleaved sequences of phonemes and durations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

20 of 27 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  6. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  7. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  8. 2020From Speech-to-Speech Translation to Automatic Dubbingcited by 25
  9. 2022VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbingcited by 20
  10. 2021Machine Translation Verbosity Control for Automatic Dubbingcited by 18
  11. 2021Isometric MT: Neural Machine Translation for Automatic Dubbingcited by 18
  12. 2022Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbingcited by 17
8 more
  1. 2017Predicting Target Language CCG Supertags Improves Neural Machine Translationcited by 15
  2. 2021Isochrony-Aware Neural Machine Translation for Automatic Dubbingcited by 14
  3. 2019Prosodic Phrase Alignment for Machine Dubbingcited by 13
  4. 2019Novel Applications of Factored Neural Machine Translationcited by 6
  5. 2022Prosodic Alignment for off-screen automatic dubbingcited by 6
  6. 2023Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbingcited by 5
  7. 2022AppTek's Submission to the IWSLT 2022 Isometric Spoken Language Translation Taskcited by 3
  8. 2021Faithful Target Attribute Prediction in Neural Machine Translationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  2. 2024Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languagescited by 0
  3. 2024Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translationcited by 0