mlsanityData to 7 Oct 2026

BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation

arXiv 2305.18326 ↗First posted 23 May 2023cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

We present a large-scale video subtitle translation dataset, BigVideo, to facilitate the study of multi-modality machine translation. Compared with the widely used How2 and VaTeX datasets, BigVideo is more than 10 times larger, consisting of 4.5 million sentence pairs and 9,981 hours of videos. We also introduce two deliberately designed test sets to verify the necessity of visual information: Ambiguous with the presence of ambiguous words, and Unambiguous in which the text context is self-contained for translation. To better model the common semantics shared across texts and videos, we introduce a contrastive learning method in the cross-modal encoder. Extensive experiments on the BigVideo show that: a) Visual information consistently improves the NMT model in terms of BLEU, BLEURT, and COMET on both Ambiguous and Unambiguous test sets. b) Visual information helps disambiguation, compared to the strong text baseline on terminology-targeted scores and human evaluation. Dataset and our implementations are available at https://github.com/DeepLearnXMU/BigVideo-VMT.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 41 of its 58 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

41 of 58 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
29 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  5. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  6. 2020Understanding the Behaviour of Contrastive Losscited by 397
  7. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  8. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  9. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  10. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  11. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  12. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  13. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  14. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  15. 2017Imagination improves Multimodal Translationcited by 38
  16. 2020Dynamic Context-guided Capsule Network for Multimodal Machine Translationcited by 36
  17. 2022On Vision Features in Multimodal Machine Translationcited by 31
  18. 2020Revisiting Round-Trip Translation for Quality Estimationcited by 24
  19. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  20. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  21. 2016Multimodal Pivots for Image Caption Translationcited by 18
  22. 2021On the Evaluation of Machine Translation for Terminology Consistencycited by 15
  23. 2020Learning to Localize Actions from Momentscited by 13
  24. 2022VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translationcited by 11
  25. 2021MultiSubs: A Large-scale Multimodal and Multilingual Datasetcited by 10
  26. 2019Cross-lingual Visual Verb Sense Disambiguationcited by 9
  27. 2022Is the Elephant Flying? Resolving Ambiguities in Text-to-Image Generative Modelscited by 7
  28. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  29. 2020Keyframe Segmentation and Positional Encoding for Video-guided Machine Translation Challenge 2020cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 2; 2026: 2.

9 papers.

  1. 2023Exploring Better Text Image Translation with Multimodal Codebookcited by 23
  2. 2024Translatotron-V(ison): An End-to-End Model for In-Image Machine Translationcited by 14
  3. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  4. 2023Video-Helpful Multimodal Machine Translationcited by 3
  5. 2023Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filteringcited by 2
  6. 2025TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentariescited by 2
  7. 2025Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completioncited by 2
  8. 2026Video-guided Machine Translation with Global Video Contextcited by 0
  9. 2026Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translationcited by 0