# BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation

arXiv: https://arxiv.org/abs/2305.18326 · HTML: https://mlsanity.com/paper/2305.18326/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data](https://mlsanity.com/paper/2001.07966/) (2020)
- [Probing the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/1903.08678/) (2019)
- [XGPT: Cross-modal Generative Pre-Training for Image Captioning](https://mlsanity.com/paper/2003.01473/) (2020)
- [A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation](https://mlsanity.com/paper/2007.08742/) (2020)
- [Mixed Precision Training of Convolutional Neural Networks using Integer Operations](https://mlsanity.com/paper/1802.00930/) (2018)
- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Imagination improves Multimodal Translation](https://mlsanity.com/paper/1705.04350/) (2017)
- [Dynamic Context-guided Capsule Network for Multimodal Machine Translation](https://mlsanity.com/paper/2009.02016/) (2020)
- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [Revisiting Round-Trip Translation for Quality Estimation](https://mlsanity.com/paper/2004.13937/) (2020)

## Built on by (most cited first, 9 of 9)

- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation](https://mlsanity.com/paper/2407.02894/) (2024)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Video-Helpful Multimodal Machine Translation](https://mlsanity.com/paper/2310.20201/) (2023)
- [Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering](https://mlsanity.com/paper/2309.04734/) (2023)
- [TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries](https://mlsanity.com/paper/2505.05714/) (2025)
- [Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion](https://mlsanity.com/paper/2508.04453/) (2025)
- [Video-guided Machine Translation with Global Video Context](https://mlsanity.com/paper/2604.06789/) (2026)
- [Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation](https://mlsanity.com/paper/2609.38660/) (2026)

## Abstract (arXiv)

We present a large-scale video subtitle translation dataset, BigVideo, to facilitate the study of multi-modality machine translation. Compared with the widely used How2 and VaTeX datasets, BigVideo is more than 10 times larger, consisting of 4.5 million sentence pairs and 9,981 hours of videos. We also introduce two deliberately designed test sets to verify the necessity of visual information: Ambiguous with the presence of ambiguous words, and Unambiguous in which the text context is self-contained for translation. To better model the common semantics shared across texts and videos, we introduce a contrastive learning method in the cross-modal encoder. Extensive experiments on the BigVideo show that: a) Visual information consistently improves the NMT model in terms of BLEU, BLEURT, and COMET on both Ambiguous and Unambiguous test sets. b) Visual information helps disambiguation, compared to the strong text baseline on terminology-targeted scores and human evaluation. Dataset and our implementations are available at https://github.com/DeepLearnXMU/BigVideo-VMT.