# VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation

arXiv: https://arxiv.org/abs/2201.08054 · HTML: https://mlsanity.com/paper/2201.08054/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 20 Jan 2022.

## Builds on (most cited first, 9 of 9)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description](https://mlsanity.com/paper/1710.07177/) (2017)
- [Probing the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/1903.08678/) (2019)
- [Multimodal Machine Translation through Visuals and Speech](https://mlsanity.com/paper/1911.12798/) (2019)
- [Keyframe Segmentation and Positional Encoding for Video-guided Machine Translation Challenge 2020](https://mlsanity.com/paper/2006.12799/) (2020)

## Built on by (most cited first, 11 of 11)

- [BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation](https://mlsanity.com/paper/2305.18326/) (2023)
- [Is the Elephant Flying? Resolving Ambiguities in Text-to-Image Generative Models](https://mlsanity.com/paper/2211.12503/) (2022)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset](https://mlsanity.com/paper/2404.18413/) (2024)
- [ODIN: A NL2SQL Recommender to Handle Schema Ambiguity](https://mlsanity.com/paper/2505.19302/) (2025)
- [Video-Helpful Multimodal Machine Translation](https://mlsanity.com/paper/2310.20201/) (2023)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)
- [A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation](https://mlsanity.com/paper/2306.07198/) (2023)
- [TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries](https://mlsanity.com/paper/2505.05714/) (2025)
- [Bi-Grid Reconstruction for Image Anomaly Detection](https://mlsanity.com/paper/2504.00609/) (2025)
- [Video-guided Machine Translation with Global Video Context](https://mlsanity.com/paper/2604.06789/) (2026)

## Abstract (arXiv)

Existing multimodal machine translation (MMT) datasets consist of images and video captions or general subtitles, which rarely contain linguistic ambiguity, making visual information not so effective to generate appropriate translations. We introduce VISA, a new dataset that consists of 40k Japanese-English parallel sentence pairs and corresponding video clips with the following key features: (1) the parallel sentences are subtitles from movies and TV episodes; (2) the source subtitles are ambiguous, which means they have multiple possible translations with different meanings; (3) we divide the dataset into Polysemy and Omission according to the cause of ambiguity. We show that VISA is challenging for the latest MMT system, and we hope that the dataset can facilitate MMT research. The VISA dataset is available at: https://github.com/ku-nlp/VISA.