mlsanityData to 14 Oct 2026

From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models

arXiv 2212.10846 ↗First posted 21 Dec 2022cs.CV cs.MMcited by 115

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answering (VQA) remains challenging, primarily due to the modality disconnection and task disconnection between LLM and VQA task. End-to-end training on vision and language data may bridge the disconnections, but is inflexible and computationally expensive. To address this issue, we propose Img2Prompt, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training. In order to provide such prompts, we further employ LLM-agnostic models to provide prompts that can describe image content and self-constructed question-answer pairs, which can effectively guide LLM to perform zero-shot VQA tasks. Img2Prompt offers the following benefits: 1) It can flexibly work with various LLMs to perform VQA. 2)~Without the needing of end-to-end training, it significantly reduces the cost of deploying LLM for zero-shot VQA tasks. 3) It achieves comparable or better performance than methods relying on end-to-end training. For example, we outperform Flamingo citeDeepmind:Flamingo2022 by 5.6% on VQAv2. On the challenging A-OKVQA dataset, our method even outperforms few-shot methods by as much as 20%.

Key facts. Cited by 115 papers in the map (data through 14 Oct 2026). Builds on 57 of its 73 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

57 of 73 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,506
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  11. 2018Hierarchical Neural Story Generationcited by 1,193
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  13. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  15. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  17. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  18. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  19. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  20. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  21. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  22. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  23. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  24. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  25. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  26. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  27. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  28. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  29. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  30. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  31. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  32. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 79
  33. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  34. 2022All You May Need for VQA are Image Captionscited by 65
  35. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  36. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 59
  37. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  38. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 37; 2024: 36; 2025: 27; 2026: 15.

115 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 909
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  3. 2023Multimodal Large Language Models: A Surveycited by 195
  4. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  5. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  6. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  7. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  8. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  9. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  10. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 69
  11. 2023Learning to Model the World with Languagecited by 68
  12. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4cited by 68
38 more
  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  2. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  3. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  4. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  5. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 54
  6. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  7. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  8. 2023SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgerycited by 48
  9. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 47
  10. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  11. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  12. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 38
  13. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  14. 2023Large Language Models Are Zero-Shot Text Classifierscited by 35
  15. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  16. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  17. 2023SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Modelscited by 30
  18. 2024Large Language Models for Captioning and Retrieving Remote Sensing Imagescited by 30
  19. 2024Efficacy of Synthetic Data as a Benchmarkcited by 30
  20. 2024Training-free Video Temporal Grounding using Large-scale Pre-trained Modelscited by 28
  21. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 25
  22. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 24
  23. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  24. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  25. 2023MuseChat: A Conversational Music Recommendation System for Videoscited by 21
  26. 2025Benchmarking Visual Language Models on Standardized Visualization Literacy Testscited by 20
  27. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  28. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  29. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  30. 2024Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Modelscited by 17
  31. 2024GP-VLS: A general-purpose vision language model for surgerycited by 17
  32. 2025Graph4MM: Weaving Multimodal Learning with Structural Informationcited by 17
  33. 2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMscited by 15
  34. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Modelscited by 15
  35. 2025CrowdVLM-R1: Expanding R1 Ability to Vision Language Model for Crowd Counting using Fuzzy Group Relative Policy Rewardcited by 15
  36. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answeringcited by 14
  37. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  38. 2023RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answeringcited by 12

50 most cited of 115 All 115 (JSON)