mlsanityData to 13 Oct 2026

VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation

arXiv 2201.08054 ↗First posted 20 Jan 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Existing multimodal machine translation (MMT) datasets consist of images and video captions or general subtitles, which rarely contain linguistic ambiguity, making visual information not so effective to generate appropriate translations. We introduce VISA, a new dataset that consists of 40k Japanese-English parallel sentence pairs and corresponding video clips with the following key features: (1) the parallel sentences are subtitles from movies and TV episodes; (2) the source subtitles are ambiguous, which means they have multiple possible translations with different meanings; (3) we divide the dataset into Polysemy and Omission according to the cause of ambiguity. We show that VISA is challenging for the latest MMT system, and we hope that the dataset can facilitate MMT research. The VISA dataset is available at: https://github.com/ku-nlp/VISA.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 9 of its 25 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

9 of 25 references.

References · 9

  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,264
  2. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,025
  3. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 413
  4. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  5. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  6. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  7. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  8. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  9. 2020Keyframe Segmentation and Positional Encoding for Video-guided Machine Translation Challenge 2020cited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 4; 2026: 1.

11 papers.

  1. 2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
  2. 2022Is the Elephant Flying? Resolving Ambiguities in Text-to-Image Generative Modelscited by 7
  3. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  4. 20243AM: An Ambiguity-Aware Multi-Modal Machine Translation Datasetcited by 4
  5. 2023Video-Helpful Multimodal Machine Translationcited by 3
  6. 2025ODIN: A NL2SQL Recommender to Handle Schema Ambiguitycited by 3
  7. 2025Trick or Neat: Adversarial Ambiguity and Language Model Evaluationcited by 3
  8. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translationcited by 2
  9. 2025TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentariescited by 2
  10. 2025Bi-Grid Reconstruction for Image Anomaly Detectioncited by 0
  11. 2026Video-guided Machine Translation with Global Video Contextcited by 0