# Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts

arXiv: https://arxiv.org/abs/2311.09050 · HTML: https://mlsanity.com/paper/2311.09050/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [KAT: A Knowledge Augmented Transformer for Vision-and-Language](https://mlsanity.com/paper/2112.08614/) (2021)
- [Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations](https://mlsanity.com/paper/2205.12685/) (2022)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [Explicit Knowledge-based Reasoning for Visual Question Answering](https://mlsanity.com/paper/1511.02570/) (2015)
- [All You May Need for VQA are Image Captions](https://mlsanity.com/paper/2205.01883/) (2022)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning](https://mlsanity.com/paper/2301.05226/) (2023)
- [MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting](https://mlsanity.com/paper/2210.07179/) (2022)
- [Modular and Parameter-Efficient Multimodal Fusion with Prompting](https://mlsanity.com/paper/2203.08055/) (2022)
- [WeaQA: Weak Supervision via Captions for Visual Question Answering](https://mlsanity.com/paper/2012.02356/) (2020)
- [Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2303.01903/) (2023)
- [MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering](https://mlsanity.com/paper/2303.01239/) (2023)

## Built on by (most cited first, 20 of 20)

- [TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding](https://mlsanity.com/paper/2404.09797/) (2024)
- [A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks](https://mlsanity.com/paper/2411.06284/) (2024)
- [Improving Generalization in Visual Reasoning via Self-Ensemble](https://mlsanity.com/paper/2410.20883/) (2024)
- [Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering](https://mlsanity.com/paper/2311.17331/) (2023)
- [Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations](https://mlsanity.com/paper/2402.12198/) (2024)
- [An LLM-Enhanced Adversarial Editing System for Lexical Simplification](https://mlsanity.com/paper/2402.14704/) (2024)
- [Benchmarking VLMs' Reasoning About Persuasive Atypical Images](https://mlsanity.com/paper/2409.10719/) (2024)
- [MirrorDiffusion: Stabilizing Diffusion Process in Zero-shot Image Translation by Prompts Redescription and Beyond](https://mlsanity.com/paper/2401.03221/) (2024)
- [Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQA](https://mlsanity.com/paper/2406.12746/) (2024)
- [Foundation Models for Autonomous Robots in Unstructured Environments](https://mlsanity.com/paper/2407.14296/) (2024)
- [Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience](https://mlsanity.com/paper/2505.05515/) (2025)
- [Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?](https://mlsanity.com/paper/2506.10912/) (2025)
- [AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering](https://mlsanity.com/paper/2407.19410/) (2024)
- [ExpliCIT-QA: Explainable Code-Based Image Table Question Answering](https://mlsanity.com/paper/2507.11694/) (2025)
- [Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning](https://mlsanity.com/paper/2608.21431/) (2026)
- [Combining Knowledge Graph and LLMs for Enhanced Zero-shot Visual Question Answering](https://mlsanity.com/paper/2501.12697/) (2025)
- [GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance](https://mlsanity.com/paper/2505.19354/) (2025)
- [Zero-shot reasoning for simulating scholarly peer-review](https://mlsanity.com/paper/2510.02027/) (2025)
- [OAD-Promoter: Enhancing Zero-shot VQA using Large Language Models with Object Attribute Description](https://mlsanity.com/paper/2511.12131/) (2025)
- [Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interaction](https://mlsanity.com/paper/2610.18161/) (2026)

## Abstract (arXiv)

Zero-shot Visual Question Answering (VQA) is a prominent vision-language task that examines both the visual and textual understanding capability of systems in the absence of training data. Recently, by converting the images into captions, information across multi-modalities is bridged and Large Language Models (LLMs) can apply their strong zero-shot generalization capability to unseen questions. To design ideal prompts for solving VQA via LLMs, several studies have explored different strategies to select or generate question-answer pairs as the exemplar prompts, which guide LLMs to answer the current questions effectively. However, they totally ignore the role of question prompts. The original questions in VQA tasks usually encounter ellipses and ambiguity which require intermediate reasoning. To this end, we present Reasoning Question Prompts for VQA tasks, which can further activate the potential of LLMs in zero-shot scenarios. Specifically, for each question, we first generate self-contained questions as reasoning question prompts via an unsupervised question edition module considering sentence fluency, semantic integrity and syntactic invariance. Each reasoning question prompt clearly indicates the intent of the original question. This results in a set of candidate answers. Then, the candidate answers associated with their confidence scores acting as answer heuristics are fed into LLMs and produce the final answer. We evaluate reasoning question prompts on three VQA challenges, experimental results demonstrate that they can significantly improve the results of LLMs on zero-shot setting and outperform existing state-of-the-art zero-shot methods on three out of four data sets. Our source code is publicly released at urlhttps://github.com/ECNU-DASE-NLP/RQP.