mlsanityData to 14 Oct 2026

A Corpus for Reasoning About Natural Language Grounded in Photographs

arXiv 1811.00491 ↗First posted 1 Nov 2018cs.CL cs.CVcited by 459

Not on the map. Why?

Cited by 150 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge.

Key facts. Cited by 459 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 70; 2022: 83; 2023: 112; 2024: 76; 2025: 72; 2026: 46.

459 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  8. 2021Transformers in Vision: A Surveycited by 1,013
  9. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  11. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
38 more
  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  3. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  4. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  5. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  6. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  7. 2024What matters when building vision-language models?cited by 326
  8. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  10. 2022Visual Spatial Reasoningcited by 293
  11. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  13. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  14. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  15. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 251
  16. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  17. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  18. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  19. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  20. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  21. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  22. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  23. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  24. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  25. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  26. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  27. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  28. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  29. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 163
  30. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  31. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  32. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 133
  33. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  34. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  35. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  36. 2021Causal Attention for Vision-Language Taskscited by 111
  37. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  38. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110

50 most cited of 459 All 459 (JSON)