mlsanityData to 14 Oct 2026

From Speech-to-Speech Translation to Automatic Dubbing

arXiv 2001.06785 ↗First posted 19 Jan 2020cs.CL cs.SD eess.AScited by 25

Not on the map. Why?

Abstract (arXiv)

We present enhancements to a speech-to-speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of the translation with the original speech segments, neural text-to-speech with fine tuning of the duration of each utterance, and, finally, audio rendering to enriches text-to-speech output with background noise and reverberation extracted from the original audio. We report on a subjective evaluation of automatic dubbing of excerpts of TED Talks from English into Italian, which measures the perceived naturalness of automatic dubbing and the relative importance of each proposed enhancement.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 5; 2024: 2; 2025: 4; 2026: 6.

25 papers.

Sort:

By area:

  1. 2021AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languagescited by 24
  2. 2022VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbingcited by 20
  3. 2021Machine Translation Verbosity Control for Automatic Dubbingcited by 18
  4. 2021Isometric MT: Neural Machine Translation for Automatic Dubbingcited by 18
  5. 2022Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbingcited by 17
  6. 2021Isochrony-Aware Neural Machine Translation for Automatic Dubbingcited by 14
  7. 2022Prosodic Alignment for off-screen automatic dubbingcited by 6
  8. 2023Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbingcited by 5
  9. 2023Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learningcited by 3
  10. 2023Improving Isochronous Machine Translation with Target Factors and Auxiliary Counterscited by 3
  11. 2025Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimizationcited by 3
  12. 2022Technology Pipeline for Large Scale Cross-Lingual Dubbing of Lecture Videos into Multiple Indian Languagescited by 2
13 more
  1. 2025Prompting LLMs: Length Control for Isometric Machine Translationcited by 2
  2. 2026Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitlingcited by 2
  3. 2024IsoChronoMeter: A simple and effective isochronic translation evaluation metriccited by 1
  4. 2025Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbingcited by 1
  5. 2026From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimizationcited by 1
  6. 2023Evaluating the Efficacy of Length-Controllable Machine Translationcited by 0
  7. 2023Efficient Spoken Language Recognition via Multilabel Classificationcited by 0
  8. 2024Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languagescited by 0
  9. 2025MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translationcited by 0
  10. 2026CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarizationcited by 0
  11. 2026PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbingcited by 0
  12. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0
  13. 2026Align Then Reason: A Multimodal Lip-Sync Judge for Dubbingcited by 0