mlsanityData to 14 Oct 2026

From Recognition to Cognition: Visual Commonsense Reasoning

arXiv 1811.10830 ↗First posted 27 Nov 2018cs.CV cs.CLcited by 556

Not on the map. Why?

Cited by 166 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and high-quality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (~45%). To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (~65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

Key facts. Cited by 556 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 83; 2022: 118; 2023: 131; 2024: 100; 2025: 87; 2026: 37.

556 papers.

Sort:

By area:

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2021Transformers in Vision: A Surveycited by 1,013
  5. 2023A Survey on Multimodal Large Language Modelscited by 909
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  9. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  11. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
38 more
  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  3. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  4. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  5. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  6. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  8. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  9. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  11. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  12. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  13. 2022Visual Spatial Reasoningcited by 293
  14. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  15. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  16. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  17. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  18. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  19. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  20. 2023Video Understanding with Large Language Models: A Surveycited by 206
  21. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  22. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  23. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  24. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  25. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  26. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  27. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  28. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  29. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  30. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  31. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  32. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  33. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  34. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  35. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  36. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  37. 2021Towards Long-Form Video Understandingcited by 146
  38. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140

50 most cited of 556 All 556 (JSON)