mlsanityData to 14 Oct 2026

A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge

arXiv 2206.01718 ↗First posted 3 Jun 2022cs.CV cs.CLcited by 710

Not on the map. Why?

Cited by 433 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The Visual Question Answering (VQA) task aspires to provide a meaningful testbed for the development of AI models that can jointly reason over visual and natural language inputs. Despite a proliferation of VQA datasets, this goal is hindered by a set of common limitations. These include a reliance on relatively simplistic questions that are repetitive in both concepts and linguistic structure, little world knowledge needed outside of the paired image, and limited reasoning required to arrive at the correct answer. We introduce A-OKVQA, a crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer. In contrast to the existing knowledge-based VQA datasets, the questions generally cannot be answered by simply querying a knowledge base, and instead require some form of commonsense reasoning about the scene depicted in the image. We demonstrate the potential of this new dataset through a detailed analysis of its contents and baseline performance measurements over a variety of state-of-the-art vision-language models. Project page: http://a-okvqa.allenai.org/

Key facts. Cited by 710 papers in the map (data through 14 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

48 of 66 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
36 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  13. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  14. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  15. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  16. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  17. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  18. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  19. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  20. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  21. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 168
  22. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  23. 2015Exploring Models and Data for Image Question Answeringcited by 139
  24. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  25. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  26. 2021WebQA: Multihop and Multimodal QAcited by 109
  27. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  28. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  29. 2017Automatic Understanding of Image and Video Advertisementscited by 68
  30. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  31. 2014Question Answering with Subgraph Embeddingscited by 58
  32. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53
  33. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  34. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  35. 2018Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captionscited by 22
  36. 2017R3: Reinforced Reader-Ranker for Open-Domain Question Answeringcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 110; 2024: 223; 2025: 217; 2026: 152.

710 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
38 more
  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  8. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  13. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  14. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  15. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  16. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  17. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  18. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  19. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  20. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  21. 2024What matters when building vision-language models?cited by 326
  22. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  23. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  24. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  25. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  26. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  27. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  28. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  29. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  30. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  31. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  32. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  33. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  34. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  35. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  36. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  37. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  38. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187

50 most cited of 710 All 710 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  2. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  3. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  4. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  5. 2025ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering
  6. 2022ChiQA: A Large Scale Image-based Real-World Question Answering Dataset for Multi-Modal Understanding
  7. 2021'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks
  8. 2025Visual question answering: from early developments to recent advances -- a survey