mlsanityData to 7 Oct 2026

VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

arXiv 1904.03493 ↗First posted 6 Apr 2019cs.CV cs.CL cs.LGcited by 412

Not on the map. Why?

Cited by 58 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEX for other video-and-language research.

Key facts. Cited by 412 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 44; 2022: 66; 2023: 84; 2024: 87; 2025: 78; 2026: 53.

412 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  7. 2023Learning Interactive Real-World Simulatorscited by 440
  8. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  9. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  12. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
38 more
  1. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  2. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  3. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  4. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  5. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  6. 2023A Survey on Video Diffusion Modelscited by 203
  7. 2023Video Understanding with Large Language Models: A Surveycited by 203
  8. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  9. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  10. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  11. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  12. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  13. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  14. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  15. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  16. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  17. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  18. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  19. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  20. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  21. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  22. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  23. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  24. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  25. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  26. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  27. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  28. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  29. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  30. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  31. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  32. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  33. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  34. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  35. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  36. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
  37. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  38. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78

50 most cited of 412 All 412 (JSON)