mlsanityData to 7 Oct 2026

PromptCap: Prompt-Guided Task-Aware Image Captioning

arXiv 2211.09699 ↗First posted 15 Nov 2022cs.CV cs.CLcited by 104

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Knowledge-based visual question answering (VQA) involves questions that require world knowledge beyond the image to yield the correct answer. Large language models (LMs) like GPT-3 are particularly helpful for this task because of their strong knowledge retrieval and reasoning capabilities. To enable LM to understand images, prior work uses a captioning model to convert images into text. However, when summarizing an image in a single caption sentence, which visual entities to describe are often underspecified. Generic image captions often miss visual details essential for the LM to answer visual questions correctly. To address this challenge, we propose PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs. Different from generic captions, PromptCap takes a natural-language prompt to control the visual entities to describe in the generated caption. The prompt contains a question that the caption should aid in answering. To avoid extra annotation, PromptCap is trained by examples synthesized with GPT-3 and existing datasets. We demonstrate PromptCap's effectiveness on an existing pipeline in which GPT-3 is prompted with image captions to carry out VQA. PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60.4% on OK-VQA and 59.6% on A-OKVQA). Zero-shot results on WebQA show that PromptCap generalizes well to unseen domains.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 83 of its 108 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

83 of 108 references.

References · 83

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  16. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  17. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  18. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  19. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  20. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  21. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  22. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  24. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  25. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  26. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  27. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  28. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  29. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  30. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  31. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  32. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  33. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  34. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  35. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  36. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  37. 2021MetaICL: Learning to Learn In Contextcited by 450
  38. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 41; 2024: 40; 2025: 15; 2026: 8.

104 papers.

Sort:

By area:

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  5. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  6. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  7. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  8. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  9. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  10. 2023Video Understanding with Large Language Models: A Surveycited by 203
  11. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  12. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
38 more
  1. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  2. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  3. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  4. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  5. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  6. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  7. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  8. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  9. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  10. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  11. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  12. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  13. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  14. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  15. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  16. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  17. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  18. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  19. 2025Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Modelscited by 34
  20. 2024Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generationcited by 28
  21. 2024SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Modelscited by 28
  22. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  23. 2023IC3: Image Captioning by Committee Consensuscited by 20
  24. 2025Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Spacecited by 20
  25. 2024Chat-Edit-3D: Interactive 3D Scene Editing via Text Promptscited by 19
  26. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 17
  27. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  28. 2024FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correctioncited by 16
  29. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  30. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 16
  31. 2024Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answeringcited by 15
  32. 2025Thinking with Visual Abstract: Enhancing Multimodal Reasoning via Visual Abstractioncited by 15
  33. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answeringcited by 14
  34. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  35. 2023A Simple Baseline for Knowledge-Based Visual Question Answeringcited by 14
  36. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  37. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  38. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoningcited by 13

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering
  2. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
  3. 2025SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering
  4. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  5. 2025Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning
  6. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts
  7. 2024Disentangling Knowledge-based and Visual Reasoning by Question Decomposition in KB-VQA
  8. 2024FlexCap: Describe Anything in Images in Controllable Detail