# WebQuest: A Benchmark for Multimodal QA on Web Page Sequences

arXiv: https://arxiv.org/abs/2409.13711 · HTML: https://mlsanity.com/paper/2409.13711/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 24 of 24)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [WebSRC: A Dataset for Web-Based Structural Reading Comprehension](https://mlsanity.com/paper/2101.09465/) (2021)
- [VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?](https://mlsanity.com/paper/2404.05955/) (2024)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://mlsanity.com/paper/2010.04295/) (2020)
- [Towards Complex Document Understanding By Discrete Reasoning](https://mlsanity.com/paper/2207.11871/) (2022)
- [ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots](https://mlsanity.com/paper/2209.08199/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [MMInA: Benchmarking Multihop Multimodal Internet Agents](https://mlsanity.com/paper/2404.09992/) (2024)

## Built on by (most cited first, 12 of 12)

- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating](https://mlsanity.com/paper/2412.18424/) (2024)
- [OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use](https://mlsanity.com/paper/2508.04482/) (2025)
- [FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow](https://mlsanity.com/paper/2505.17399/) (2025)
- [WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation](https://mlsanity.com/paper/2508.16763/) (2025)
- [IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?](https://mlsanity.com/paper/2509.24709/) (2025)
- [DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness](https://mlsanity.com/paper/2511.01323/) (2025)
- [Same Content, Different Representations: A Controlled Study for Table QA](https://mlsanity.com/paper/2509.22983/) (2025)
- [Composition-Grounded Data Synthesis for Visual Reasoning](https://mlsanity.com/paper/2510.15040/) (2025)
- [Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge](https://mlsanity.com/paper/2511.00505/) (2025)
- [GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection](https://mlsanity.com/paper/2512.09396/) (2025)
- [MemoWM: How World Models Change What Agents Need to Remember](https://mlsanity.com/paper/2610.10778/) (2026)

## Abstract (arXiv)

The rise of powerful multimodal LLMs has enhanced the viability of building web agents which can, with increasing levels of autonomy, assist users to retrieve information and complete tasks on various human-computer interfaces. It is hence necessary to build challenging benchmarks that span a wide-variety of use cases reflecting real-world usage. In this work, we present WebQuest, a multi-page question-answering dataset that requires reasoning across multiple related web pages. In contrast to existing UI benchmarks that focus on multi-step web navigation and task completion, our dataset evaluates information extraction, multimodal retrieval and composition of information from many web pages. WebQuest includes three question categories: single-screen QA, multi-screen QA, and QA based on navigation traces. We evaluate leading proprietary multimodal models like GPT-4V, Gemini Flash, Claude 3, and open source models like InstructBLIP, PaliGemma on our dataset, revealing a significant gap between single-screen and multi-screen reasoning. Finally, we investigate inference time techniques like Chain-of-Thought prompting to improve model capabilities on multi-screen reasoning.