mlsanityData to 14 Oct 2026

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

arXiv 2205.12522 ↗First posted 25 May 2022cs.CV cs.CLcited by 100

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show superior correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

18 of 32 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  7. 2021Scaling Vision Transformerscited by 999
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2016Self-critical Sequence Training for Image Captioningcited by 547
  11. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  12. 2019Connecting Vision and Language with Localized Narrativescited by 226
6 more
  1. 2020BLEU might be Guilty but References are not Innocentcited by 110
  2. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  3. 2018Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translationcited by 58
  4. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  5. 2020Clue: Cross-modal Coherence Modeling for Caption Generationcited by 24
  6. 2020Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coveragecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 17; 2024: 28; 2025: 35; 2026: 16.

100 papers.

Sort:

By area:

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  3. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  4. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  5. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  6. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  7. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 101
  8. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  9. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 76
  10. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  11. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  12. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
38 more
  1. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55
  2. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  3. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  4. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  5. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  6. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 37
  7. 2023AltDiffusion: A Multilingual Text-to-Image Diffusion Modelcited by 30
  8. 2022Geographic and Geopolitical Biases of Language Modelscited by 29
  9. 2023Retrieval-augmented Image Captioningcited by 29
  10. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Taskscited by 27
  11. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  12. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generationcited by 25
  13. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 25
  14. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  15. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24
  16. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 22
  17. 2024Fairness and Bias in Multimodal AI: A Surveycited by 20
  18. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  19. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  20. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  21. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  22. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledgecited by 16
  23. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 15
  24. 2025MIEB: Massive Image Embedding Benchmarkcited by 15
  25. 2024Analyzing The Language of Visual Tokenscited by 12
  26. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  27. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11
  28. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  29. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  30. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  31. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  32. 2025Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systemscited by 10
  33. 2022MaXM: Towards Multilingual Visual Question Answeringcited by 9
  34. 2024mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpuscited by 9
  35. 2025Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Modelscited by 9
  36. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  37. 2023Several categories of Large Language Models (LLMs): A Short Surveycited by 7
  38. 2025CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Modelscited by 7

50 most cited of 100 All 100 (JSON)