mlsanityData to 7 Oct 2026

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA

arXiv 2109.05014 ↗First posted 10 Sep 2021cs.CVcited by 298

Not on the map. Why?

Cited by 86 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge-based visual question answering (VQA) involves answering questions that require external knowledge not present in the image. Existing methods first retrieve knowledge from external resources, then reason over the selected knowledge, the input image, and question for answer prediction. However, this two-step approach could lead to mismatches that potentially limit the VQA performance. For example, the retrieved knowledge might be noisy and irrelevant to the question, and the re-embedded knowledge features during reasoning might deviate from their original meanings in the knowledge base (KB). To address this challenge, we propose PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA. Inspired by GPT-3's power in knowledge retrieval and question answering, instead of using structured KBs as in previous work, we treat GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge. Specifically, we first convert the image into captions (or tags) that GPT-3 can understand, then adapt GPT-3 to solve the VQA task in a few-shot manner by just providing a few in-context VQA examples. We further boost performance by carefully investigating: (i) what text formats best describe the image content, and (ii) how in-context examples can be better selected and used. PICa unlocks the first use of GPT-3 for multimodal tasks. By using only 16 examples, PICa surpasses the supervised state of the art by an absolute +8.6 points on the OK-VQA dataset. We also benchmark PICa on VQAv2, where PICa also shows a decent few-shot performance.

Key facts. Cited by 298 papers in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

27 of 43 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
15 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  5. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  6. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  7. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  8. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  9. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  10. 2020Language Models are Open Knowledge Graphscited by 95
  11. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  12. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  13. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  14. 2018VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questionscited by 50
  15. 2018Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answeringcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 39; 2023: 117; 2024: 76; 2025: 39; 2026: 20.

298 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  10. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  11. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
38 more
  1. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  2. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  3. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  4. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  5. 2022Visual Classification via Description from Large Language Modelscited by 331
  6. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  8. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  9. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  10. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  11. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  12. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  13. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  14. 2023Multimodal Large Language Models: A Surveycited by 193
  15. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  16. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  17. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  18. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  19. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  20. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  21. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  22. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  23. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  24. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  25. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  26. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  27. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  28. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  29. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  30. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  31. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  32. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  33. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  34. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  35. 2023Can ChatGPT Assess Human Personalities? A General Evaluation Frameworkcited by 69
  36. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  37. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  38. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66

50 most cited of 298 All 298 (JSON)