# Visual Question Answering on Image Sets

arXiv: https://arxiv.org/abs/2008.11976 · HTML: https://mlsanity.com/paper/2008.11976/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2020.

## Built on by (most cited first, 26 of 26)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning](https://mlsanity.com/paper/2311.18799/) (2023)
- [Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions](https://mlsanity.com/paper/2308.04152/) (2023)
- [Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark](https://mlsanity.com/paper/2407.13766/) (2024)
- [3D Question Answering](https://mlsanity.com/paper/2112.08359/) (2021)
- [Image Retrieval from Contextual Descriptions](https://mlsanity.com/paper/2203.15867/) (2022)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.05187/) (2021)
- [VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use](https://mlsanity.com/paper/2410.16400/) (2024)
- [Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents](https://mlsanity.com/paper/2411.16740/) (2024)
- [Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning](https://mlsanity.com/paper/2212.10278/) (2022)
- [Step Differences in Instructional Video](https://mlsanity.com/paper/2404.16222/) (2024)
- [Answer Mining from a Pool of Images: Towards Retrieval-Based Visual Question Answering](https://mlsanity.com/paper/2306.16713/) (2023)
- [JDocQA: Japanese Document Question Answering Dataset for Generative Language Models](https://mlsanity.com/paper/2403.19454/) (2024)
- [ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task](https://mlsanity.com/paper/2503.04444/) (2025)
- [Co-VisiON: Co-Visibility ReasONing on Sparse Image Sets of Indoor Scenes](https://mlsanity.com/paper/2506.16805/) (2025)
- [Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMs](https://mlsanity.com/paper/2508.17334/) (2025)
- [Analysis on Image Set Visual Question Answering](https://mlsanity.com/paper/2104.00107/) (2021)
- [Audiopedia: Audio QA with Knowledge](https://mlsanity.com/paper/2412.20619/) (2024)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D](https://mlsanity.com/paper/2506.01275/) (2025)
- [Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos](https://mlsanity.com/paper/2601.16333/) (2026)
- [Instruct-ICL: Instruction-Guided In-Context Learning for Post-Disaster Damage Assessment](https://mlsanity.com/paper/2605.11439/) (2026)

## Abstract (arXiv)

We introduce the task of Image-Set Visual Question Answering (ISVQA), which generalizes the commonly studied single-image VQA problem to multi-image settings. Taking a natural language question and a set of images as input, it aims to answer the question based on the content of the images. The questions can be about objects and relationships in one or more images or about the entire scene depicted by the image set. To enable research in this new topic, we introduce two ISVQA datasets - indoor and outdoor scenes. They simulate the real-world scenarios of indoor image collections and multiple car-mounted cameras, respectively. The indoor-scene dataset contains 91,479 human annotated questions for 48,138 image sets, and the outdoor-scene dataset has 49,617 questions for 12,746 image sets. We analyze the properties of the two datasets, including question-and-answer distributions, types of questions, biases in dataset, and question-image dependencies. We also build new baseline models to investigate new research challenges in ISVQA.