mlsanityData to 7 Oct 2026

VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

arXiv 2106.04632 ↗First posted 8 Jun 2021cs.CV cs.CLcited by 95

Not on the map. Why?

Abstract (arXiv)

Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 56 of its 78 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

56 of 78 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  7. 2017Dense-Captioning Events in Videoscited by 789
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  10. 2017Localizing Moments in Video with Natural Languagecited by 660
  11. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  13. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  14. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  15. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  16. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  17. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  18. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  19. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  20. 2020Multi-modal Transformer for Video Retrievalcited by 395
  21. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  22. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  23. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  24. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  25. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  26. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  27. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  28. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  29. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  30. 2018Scaling Neural Machine Translationcited by 204
  31. 2015A Dataset for Movie Descriptioncited by 195
  32. 2020Support-set bottlenecks for video-text representation learningcited by 170
  33. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  34. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  35. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  36. 2020What is More Likely to Happen Next? Video-and-Language Future Event Predictioncited by 61
  37. 2020On Learning Universal Representations Across Languagescited by 58
  38. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 28; 2023: 20; 2024: 22; 2025: 13.

95 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  3. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  6. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  7. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  8. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  9. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  10. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  11. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  12. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
38 more
  1. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  2. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  3. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  4. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  5. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  6. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  7. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  8. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  9. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  10. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  11. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  12. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  13. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  14. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  15. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  16. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  17. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  18. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  19. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  20. 2021Co-training Transformer with Videos and Images Improves Action Recognitioncited by 41
  21. 2024Temporal Reasoning Transfer from Text to Videocited by 36
  22. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  23. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  24. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  25. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  26. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  27. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 27
  28. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  29. 2022i-Code: An Integrative and Composable Multimodal Learning Frameworkcited by 24
  30. 2022Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learningcited by 22
  31. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21
  32. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  33. 2024Narrative Action Evaluation with Prompt-Guided Multimodal Interactioncited by 19
  34. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  35. 2025Principled Multimodal Representation Learningcited by 19
  36. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  37. 2022Boundary-aware Self-supervised Learning for Video Scene Segmentationcited by 17
  38. 2024HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Textscited by 17

50 most cited of 95 All 95 (JSON)