# Towards Visually-Guided Movie Subtitle Translation for Indic Languages

arXiv: https://arxiv.org/abs/2605.11993 · HTML: https://mlsanity.com/paper/2605.11993/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 17 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 5 of 5)

- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning](https://mlsanity.com/paper/2407.03076/) (2024)
- [Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages](https://mlsanity.com/paper/2308.16075/) (2023)

## Abstract (arXiv)

Movie subtitle translation is inherently multimodal, yet text-only systems often miss visual cues needed to convey emotion, action, and social nuance, especially for low-resource Indic languages (English to Hindi, Bengali, Telugu, Tamil and Kannada). We present a case study on five full-length films and compare two lightweight visual grounding strategies: structured attribute summaries from a 5-minute sliding window and free-text summaries of inter-subtitle visual gaps. Our analysis shows that temporal misalignment between subtitles and frames is a major obstacle in long-form video, often rendering indiscriminate visual grounding ineffective. However, oracle selective grounding, which replaces only the lowest-quality 20-30% of baseline segments with visual-enhanced outputs, consistently improves COMET over the text-only baseline while requiring far less visual processing. Among the two approaches, coarse attribute-based visual context summarization is more robust, capturing scene-level emotion and contextual subtle cues that text alone often misses