mlsanityData to 14 Oct 2026

WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

arXiv 2103.01913 ↗First posted 2 Mar 2021cs.CV cs.CL cs.IRcited by 287

Not on the map. Why?

Cited by 76 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The milestone improvements brought about by deep representation learning and pre-training techniques have led to large performance gains across downstream NLP, IR and Vision tasks. Multimodal modeling techniques aim to leverage large high-quality visio-linguistic datasets for learning complementary information (across image and text modalities). In this paper, we introduce the Wikipedia-based Image Text (WIT) Dataset (https://github.com/google-research-datasets/wit) to better facilitate multimodal, multilingual learning. WIT is composed of a curated set of 37.6 million entity rich image-text examples with 11.5 million unique images across 108 Wikipedia languages. Its size enables WIT to be used as a pretraining dataset for multimodal models, as we show when applied to downstream tasks such as image-text retrieval. WIT has four main and unique advantages. First, WIT is the largest multimodal dataset by the number of image-text examples by 3x (at the time of writing). Second, WIT is massively multilingual (first of its kind) with coverage over 100+ languages (each of which has at least 12K examples) and provides cross-lingual texts for many images. Third, WIT represents a more diverse set of concepts and real world entities relative to what previous datasets cover. Lastly, WIT provides a very challenging real-world test set, as we empirically illustrate using an image-text retrieval task as an example.

Key facts. Cited by 287 papers in the map (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

33 of 44 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
21 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  10. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  11. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  12. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  13. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  14. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  15. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  16. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  17. 2018A Visual Attention Grounding Neural Model for Multimodal Machine Translationcited by 22
  18. 2016Multimodal Pivots for Image Caption Translationcited by 18
  19. 2019Graph-RISE: Graph-Regularized Image Semantic Embeddingcited by 15
  20. 2020MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and Modalitiescited by 10
  21. 2019Learning Multilingual Word Embeddings Using Image-Text Datacited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 44; 2023: 57; 2024: 73; 2025: 66; 2026: 29.

287 papers.

Sort:

By area:

  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  8. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  9. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  10. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  11. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  12. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
38 more
  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  2. 2024The Platonic Representation Hypothesiscited by 382
  3. 2024What matters when building vision-language models?cited by 326
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  7. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  8. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  9. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  10. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  11. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  12. 2023Language-Driven Representation Learning for Roboticscited by 165
  13. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  14. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  15. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 151
  16. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  17. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  18. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 137
  19. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  20. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  21. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  22. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 123
  23. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120
  24. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  25. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  26. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  27. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  28. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  29. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  30. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  31. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 90
  32. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  33. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  34. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  35. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  36. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  37. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  38. 2025Scaling Language-Free Visual Representation Learningcited by 66

50 most cited of 287 All 287 (JSON)