# WeaQA: Weak Supervision via Captions for Visual Question Answering

arXiv: https://arxiv.org/abs/2012.02356 · HTML: https://mlsanity.com/paper/2012.02356/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2020.

## Built on by (most cited first, 25 of 25)

- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)
- [All You May Need for VQA are Image Captions](https://mlsanity.com/paper/2205.01883/) (2022)
- [Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering](https://mlsanity.com/paper/2109.04014/) (2021)
- [MedCoT: Medical Chain of Thought via Hierarchical Expert](https://mlsanity.com/paper/2412.13736/) (2024)
- [Learning to Answer Visual Questions from Web Videos](https://mlsanity.com/paper/2205.05019/) (2022)
- [Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2112.10006/) (2021)
- [R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest](https://mlsanity.com/paper/2410.20327/) (2024)
- [Improving Biomedical Information Retrieval with Neural Retrievers](https://mlsanity.com/paper/2201.07745/) (2022)
- [Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts](https://mlsanity.com/paper/2311.09050/) (2023)
- [Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!](https://mlsanity.com/paper/2306.03932/) (2023)
- [Weakly Supervised Relative Spatial Reasoning for Visual Question Answering](https://mlsanity.com/paper/2109.01934/) (2021)
- [Exploring Question Decomposition for Zero-Shot VQA](https://mlsanity.com/paper/2310.17050/) (2023)
- [RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2412.02402/) (2024)
- [MaXM: Towards Multilingual Visual Question Answering](https://mlsanity.com/paper/2209.05401/) (2022)
- [CIC: A Framework for Culturally-Aware Image Captioning](https://mlsanity.com/paper/2402.05374/) (2024)
- [Unsupervised Natural Language Inference Using PHL Triplet Generation](https://mlsanity.com/paper/2110.08438/) (2021)
- [Language bias in Visual Question Answering: A Survey and Taxonomy](https://mlsanity.com/paper/2111.08531/) (2021)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding](https://mlsanity.com/paper/2309.00215/) (2023)
- [SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner](https://mlsanity.com/paper/2503.04858/) (2025)
- [SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering](https://mlsanity.com/paper/2509.20871/) (2025)
- [Weakly Supervised Visual Question Answer Generation](https://mlsanity.com/paper/2306.06622/) (2023)
- [Tackling VQA with Pretrained Foundation Models without Further Training](https://mlsanity.com/paper/2309.15487/) (2023)
- [When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs](https://mlsanity.com/paper/2509.16633/) (2025)

## Abstract (arXiv)

Methodologies for training visual question answering (VQA) models assume the availability of datasets with human-annotated textitImage-Question-Answer (I-Q-A) triplets. This has led to heavy reliance on datasets and a lack of generalization to new types of questions and scenes. Linguistic priors along with biases and errors due to annotator subjectivity have been shown to percolate into VQA models trained on such samples. We study whether models can be trained without any human-annotated Q-A pairs, but only with images and their associated textual descriptions or captions. We present a method to train models with synthetic Q-A pairs generated procedurally from captions. Additionally, we demonstrate the efficacy of spatial-pyramid image patches as a simple but effective alternative to dense and costly object bounding box annotations used in existing VQA models. Our experiments on three VQA benchmarks demonstrate the efficacy of this weakly-supervised approach, especially on the VQA-CP challenge, which tests performance under changing linguistic priors.