mlsanityData to 7 Oct 2026

GEM: A General Evaluation Benchmark for Multimodal Tasks

arXiv 2106.09889 ↗First posted 18 Jun 2021cs.CL cs.CV cs.MMcited by 16

Not on the map. Why?

Abstract (arXiv)

In this paper, we present GEM as a General Evaluation benchmark for Multimodal tasks. Different from existing datasets such as GLUE, SuperGLUE, XGLUE and XTREME that mainly focus on natural language tasks, GEM is a large-scale vision-language benchmark, which consists of GEM-I for image-language tasks and GEM-V for video-language tasks. Comparing with existing multimodal datasets such as MSCOCO and Flicker30K for image-language tasks, YouCook2 and MSR-VTT for video-language tasks, GEM is not only the largest vision-language dataset covering image-language tasks and video-language tasks at the same time, but also labeled in multiple languages. We also provide two baseline models for this benchmark. We will release the dataset, code and baseline models, aiming to advance the development of multilingual multimodal research.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

25 of 34 references.

References · 25

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  10. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
13 more
  1. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  2. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  3. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  4. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  5. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  6. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  7. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  8. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  9. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  10. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  11. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  12. 2020GLGE: A New General Language Generation Evaluation Benchmarkcited by 47
  13. 2019Learning Video Representations using Contrastive Bidirectional Transformercited by 39

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 3; 2024: 6; 2025: 2.

16 papers.

Sort:

By area:

  1. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  2. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  3. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  4. 2024DevBench: A multimodal developmental benchmark for language learningcited by 13
  5. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
  6. 2022Delving Deeper into Cross-lingual Visual Question Answeringcited by 8
  7. 2024CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answeringcited by 4
  8. 2022C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrievalcited by 3
  9. 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Modelscited by 3
  10. 2024The Use of Multimodal Large Language Models to Detect Objects from Thermal Images: Transportation Applicationscited by 2
  11. 2024@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technologycited by 2
  12. 2023Schema-Guided Semantic Accuracy: Faithfulness in Task-Oriented Dialogue Response Generationcited by 1
4 more
  1. 2024VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Taskscited by 1
  2. 2023Requirement Formalisation using Natural Language Processing and Machine Learning: A Systematic Reviewcited by 0
  3. 2023VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialoguecited by 0
  4. 2025Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generationcited by 0