# TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries

arXiv: https://arxiv.org/abs/2505.05714 · HTML: https://mlsanity.com/paper/2505.05714/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 9 May 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description](https://mlsanity.com/paper/1710.07177/) (2017)
- [Probing the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/1903.08678/) (2019)
- [Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://mlsanity.com/paper/2308.12038/) (2023)
- [A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation](https://mlsanity.com/paper/2007.08742/) (2020)
- [Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/2105.14462/) (2021)
- [Imagination improves Multimodal Translation](https://mlsanity.com/paper/1705.04350/) (2017)
- [Dynamic Context-guided Capsule Network for Multimodal Machine Translation](https://mlsanity.com/paper/2009.02016/) (2020)
- [On Vision Features in Multimodal Machine Translation](https://mlsanity.com/paper/2203.09173/) (2022)
- [Exploring Better Text Image Translation with Multimodal Codebook](https://mlsanity.com/paper/2305.17415/) (2023)
- [Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models](https://mlsanity.com/paper/2109.03415/) (2021)
- [Multimodal Pivots for Image Caption Translation](https://mlsanity.com/paper/1601.03916/) (2016)
- [VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation](https://mlsanity.com/paper/2201.08054/) (2022)
- [BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation](https://mlsanity.com/paper/2305.18326/) (2023)
- [Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation](https://mlsanity.com/paper/2212.10313/) (2022)
- [Multimodal Neural Machine Translation with Search Engine Based Image Retrieval](https://mlsanity.com/paper/2208.00767/) (2022)
- [Video-Helpful Multimodal Machine Translation](https://mlsanity.com/paper/2310.20201/) (2023)

## Built on by (most cited first, 2 of 2)

- [ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning](https://mlsanity.com/paper/2507.07306/) (2025)
- [Video-guided Machine Translation with Global Video Context](https://mlsanity.com/paper/2604.06789/) (2026)

## Abstract (arXiv)

Most existing multimodal machine translation (MMT) datasets are predominantly composed of static images or short video clips, lacking extensive video data across diverse domains and topics. As a result, they fail to meet the demands of real-world MMT tasks, such as documentary translation. In this study, we developed TopicVD, a topic-based dataset for video-supported multimodal machine translation of documentaries, aiming to advance research in this field. We collected video-subtitle pairs from documentaries and categorized them into eight topics, such as economy and nature, to facilitate research on domain adaptation in video-guided MMT. Additionally, we preserved their contextual information to support research on leveraging the global context of documentaries in video-guided MMT. To better capture the shared semantics between text and video, we propose an MMT model based on a cross-modal bidirectional attention module. Extensive experiments on the TopicVD dataset demonstrate that visual information consistently improves the performance of the NMT model in documentary translation. However, the MMT model's performance significantly declines in out-of-domain scenarios, highlighting the need for effective domain adaptation methods. Additionally, experiments demonstrate that global context can effectively improve translation performance. % Dataset and our implementations are available at https://github.com/JinzeLv/TopicVD