mlsanityData to 7 Oct 2026

Zero-shot Visual Question Answering with Language Model Feedback

arXiv 2305.17006 ↗First posted 26 May 2023cs.CV cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel language model guided captioning approach, LAMOC, for knowledge-based visual question answering (VQA). Our approach employs the generated captions by a captioning model as the context of an answer prediction model, which is a Pre-trained Language model (PLM). As the major contribution, we leverage the guidance and feedback of the prediction model to improve the capability of the captioning model. In this way, the captioning model can become aware of the task goal and information need from the PLM. To develop our approach, we design two specific training stages, where the first stage adapts the captioning model to the prediction model (selecting more suitable caption propositions for training) and the second stage tunes the captioning model according to the task goal (learning from feedback of the PLM). Extensive experiments demonstrate the effectiveness of the proposed approach on the knowledge-based VQA task. Specifically, on the challenging A-OKVQA dataset, LAMOC outperforms several competitive zero-shot methods and even achieves comparable results to a fine-tuned VLP model. Our code is publicly available at https://github.com/RUCAIBox/LAMOC.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 37 of its 43 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

37 of 43 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
25 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2018Hierarchical Neural Story Generationcited by 1,182
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  7. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  8. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  9. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  10. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  11. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  12. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  13. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  14. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  15. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  16. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  17. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  18. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  19. 2020Text Generation by Learning from Demonstrationscited by 64
  20. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  21. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  22. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  23. 2022Training Language Models with Language Feedbackcited by 48
  24. 2022Compilable Neural Code Generation with Compiler Feedbackcited by 46
  25. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 2; 2026: 2.

11 papers.

  1. 2024Improving Generalization in Visual Reasoning via Self-Ensemblecited by 15
  2. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questionscited by 14
  3. 2024Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Modelscited by 8
  4. 2024Multi-Modal Data Exploration via Language Agentscited by 7
  5. 2023In-context Interference in Chat-based Large Language Modelscited by 3
  6. 2026Retrieving Counterfactuals Improves Visual In-Context Learningcited by 3
  7. 2024Zero-Shot End-To-End Spoken Question Answering In Medical Domaincited by 2
  8. 2024LLM-based Hierarchical Concept Decomposition for Interpretable Fine-Grained Image Classificationcited by 1
  9. 2025Clarification as Supervision: Reinforcement Learning for Vision-Language Interfacescited by 1
  10. 2025GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performancecited by 0
  11. 2026Boosting Knowledge-based Visual Question Answering with Structured Context Reasoningcited by 0