mlsanityData to 14 Oct 2026

Length Aware Speech Translation for Video Dubbing

arXiv 2506.00740 ↗First posted 31 May 2025cs.CL cs.AI cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

In video dubbing, aligning translated audio with the source audio is a significant challenge. Our focus is on achieving this efficiently, tailored for real-time, on-device video dubbing scenarios. We developed a phoneme-based end-to-end length-sensitive speech translation (LSST) model, which generates translations of varying lengths short, normal, and long using predefined tags. Additionally, we introduced length-aware beam search (LABS), an efficient approach to generate translations of different lengths in a single decoding pass. This approach maintained comparable BLEU scores compared to a baseline without length awareness while significantly enhancing synchronization quality between source and target audio, achieving a mean opinion score (MOS) gain of 0.34 for Spanish and 0.65 for Korean, respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

13 of 20 references.

References · 13

  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  5. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  6. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  7. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  8. 2020Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?cited by 21
  9. 2022VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbingcited by 20
  10. 2021Machine Translation Verbosity Control for Automatic Dubbingcited by 18
  11. 2021Isochrony-Aware Neural Machine Translation for Automatic Dubbingcited by 14
  12. 2020Adapting End-to-End Speech Recognition for Readable Subtitlescited by 10
1 more
  1. 2024Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learningcited by 1