mlsanityData to 7 Oct 2026

Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training

arXiv 2210.08773 ↗First posted 17 Oct 2022cs.CVcited by 88

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PNP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering. Surpassing end-to-end trained baselines, PNP-VQA achieves state-of-the-art results on zero-shot VQAv2 and GQA. With 11B parameters, it outperforms the 80B-parameter Flamingo model by 8.5% on VQAv2. With 738M PLM parameters, PNP-VQA achieves an improvement of 9.1% on GQA over FewVLM with 740M PLM parameters. Code is released at https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

48 of 57 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
36 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  12. 2018Hierarchical Neural Story Generationcited by 1,182
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  14. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  16. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  17. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  18. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  19. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  20. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  21. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  22. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  23. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  24. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  25. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  26. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  27. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  28. 2018Language GANs Falling Shortcited by 97
  29. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  30. 2017Learning with Latent Languagecited by 75
  31. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  32. 2022All You May Need for VQA are Image Captionscited by 65
  33. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  34. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  35. 2022Natural Language Deduction through Search over Statement Compositionscited by 35
  36. 2020WeaQA: Weak Supervision via Captions for Visual Question Answeringcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 31; 2024: 35; 2025: 15; 2026: 5.

88 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  5. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  6. 2023Multimodal Large Language Models: A Surveycited by 193
  7. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  8. 2023Vision-Language Models as Success Detectorscited by 138
  9. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  10. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  11. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  12. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
38 more
  1. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  2. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  4. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  5. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  6. 2023Modular Visual Question Answering via Code Generationcited by 47
  7. 2024MedCoT: Medical Chain of Thought via Hierarchical Expertcited by 46
  8. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  9. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  10. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  11. 2024Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generationcited by 28
  12. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  13. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMscited by 25
  14. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  15. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  16. 2023IC3: Image Captioning by Committee Consensuscited by 20
  17. 2025Harnessing Large Language Models for Disaster Management: A Surveycited by 20
  18. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  19. 2024Video Anomaly Detection in 10 Years: A Survey and Outlookcited by 18
  20. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  21. 2024HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoningcited by 17
  22. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Modelscited by 15
  23. 2023A Simple Baseline for Knowledge-Based Visual Question Answeringcited by 14
  24. 2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMscited by 14
  25. 2024MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationalecited by 14
  26. 2024X-Former: Unifying Contrastive and Reconstruction Learning for MLLMscited by 14
  27. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  28. 2024Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Trainingcited by 12
  29. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  30. 2024Visual Robustness Benchmark for Visual Question Answering (VQA)cited by 12
  31. 2024Rethinking VLMs and LLMs for Image Classificationcited by 12
  32. 2025FREE: Fast and Robust Vision Language Models with Early Exitscited by 12
  33. 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
  34. 2023Can Language Models Laugh at YouTube Short-form Videos?cited by 11
  35. 2023Unleashing the Potential of Large Language Model: Zero-shot VQA for Flood Disaster Scenariocited by 10
  36. 2023Few-shot Action Recognition with Captioning Foundation Modelscited by 9
  37. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answeringcited by 9
  38. 2024Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Modelscited by 8

50 most cited of 88 All 88 (JSON)