# Length Aware Speech Translation for Video Dubbing

arXiv: https://arxiv.org/abs/2506.00740 · HTML: https://mlsanity.com/paper/2506.00740/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 13 of 13)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1811.02050/) (2018)
- [Large-Scale Streaming End-to-End Speech Translation with Neural Transducers](https://mlsanity.com/paper/2204.05352/) (2022)
- [Revisiting End-to-End Speech-to-Text Translation From Scratch](https://mlsanity.com/paper/2206.04571/) (2022)
- [Is 42 the Answer to Everything in Subtitling-oriented Speech Translation?](https://mlsanity.com/paper/2006.01080/) (2020)
- [VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing](https://mlsanity.com/paper/2211.16934/) (2022)
- [Machine Translation Verbosity Control for Automatic Dubbing](https://mlsanity.com/paper/2110.03847/) (2021)
- [Isochrony-Aware Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08548/) (2021)
- [Adapting End-to-End Speech Recognition for Readable Subtitles](https://mlsanity.com/paper/2005.12143/) (2020)
- [Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning](https://mlsanity.com/paper/2403.15469/) (2024)

## Built on by (most cited first, 1 of 1)

- [HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning](https://mlsanity.com/paper/2601.10187/) (2026)

## Abstract (arXiv)

In video dubbing, aligning translated audio with the source audio is a significant challenge. Our focus is on achieving this efficiently, tailored for real-time, on-device video dubbing scenarios. We developed a phoneme-based end-to-end length-sensitive speech translation (LSST) model, which generates translations of varying lengths short, normal, and long using predefined tags. Additionally, we introduced length-aware beam search (LABS), an efficient approach to generate translations of different lengths in a single decoding pass. This approach maintained comparable BLEU scores compared to a baseline without length awareness while significantly enhancing synchronization quality between source and target audio, achieving a mean opinion score (MOS) gain of 0.34 for Spanish and 0.65 for Korean, respectively.