# MTVR: Multilingual Moment Retrieval in Videos

arXiv: https://arxiv.org/abs/2108.00061 · HTML: https://mlsanity.com/paper/2108.00061/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [ExCL: Extractive Clip Localization Using Natural Language Descriptions](https://mlsanity.com/paper/1904.02755/) (2019)
- [Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering](https://mlsanity.com/paper/1505.05612/) (2015)
- [COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval](https://mlsanity.com/paper/1805.08661/) (2018)
- [Fluency-Guided Cross-Lingual Image Captioning](https://mlsanity.com/paper/1708.04390/) (2017)
- [Parameter Sharing Methods for Multilingual Self-Attentional Translation Models](https://mlsanity.com/paper/1809.00252/) (2018)
- [MULE: Multimodal Universal Language Embedding](https://mlsanity.com/paper/1909.03493/) (2019)
- [Learning to Scale Multilingual Representations for Vision-Language Tasks](https://mlsanity.com/paper/2004.04312/) (2020)
- [Multilingual Visual Sentiment Concept Matching](https://mlsanity.com/paper/1606.02276/) (2016)

## Built on by (most cited first, 8 of 8)

- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrieval](https://mlsanity.com/paper/2309.05451/) (2023)
- [MultiVENT: Multilingual Videos of Events with Aligned Natural Text](https://mlsanity.com/paper/2307.03153/) (2023)
- [C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval](https://mlsanity.com/paper/2210.03625/) (2022)
- [Cross-Modal Retrieval Meets Inference:Improving Zero-Shot Classification with Cross-Modal Retrieval](https://mlsanity.com/paper/2308.15273/) (2023)

## Abstract (arXiv)

We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21.8K TV show video clips. The dataset is collected by extending the popular TVR dataset (in English) with paired Chinese queries and subtitles. Compared to existing moment retrieval datasets, mTVR is multilingual, larger, and comes with diverse annotations. We further propose mXML, a multilingual moment retrieval model that learns and operates on data from both languages, via encoder parameter sharing and language neighborhood constraints. We demonstrate the effectiveness of mXML on the newly collected MTVR dataset, where mXML outperforms strong monolingual baselines while using fewer parameters. In addition, we also provide detailed dataset analyses and model ablations. Data and code are publicly available at https://github.com/jayleicn/mTVRetrieval