# MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

arXiv: https://arxiv.org/abs/2508.13186 · HTML: https://mlsanity.com/paper/2508.13186/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 37)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension](https://mlsanity.com/paper/2404.16790/) (2024)
- [BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese](https://mlsanity.com/paper/2504.19314/) (2025)
- [OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation](https://mlsanity.com/paper/2505.23885/) (2025)
- [MMSearch-R1: Incentivizing LMMs to Search](https://mlsanity.com/paper/2506.20670/) (2025)

## Built on by (most cited first, 30 of 42)

- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents](https://mlsanity.com/paper/2602.14234/) (2026)
- [Benchmark Test-Time Scaling of General LLM Agents](https://mlsanity.com/paper/2602.18998/) (2026)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications](https://mlsanity.com/paper/2510.16724/) (2025)
- [OmniGAIA: Towards Native Omni-Modal AI Agents](https://mlsanity.com/paper/2602.22897/) (2026)
- [MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline](https://mlsanity.com/paper/2603.01050/) (2026)
- [VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking](https://mlsanity.com/paper/2603.20185/) (2026)
- [Reinforcement Learning Foundations for Deep Research Systems: A Survey](https://mlsanity.com/paper/2509.06733/) (2025)
- [VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents](https://mlsanity.com/paper/2603.16289/) (2026)
- [InteractComp: Evaluating Search Agents With Ambiguous Queries](https://mlsanity.com/paper/2510.24668/) (2025)
- [Towards Long-horizon Agentic Multimodal Search](https://mlsanity.com/paper/2604.12890/) (2026)
- [ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents](https://mlsanity.com/paper/2605.12481/) (2026)
- [BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents](https://mlsanity.com/paper/2602.12876/) (2026)
- [Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning](https://mlsanity.com/paper/2606.15231/) (2026)
- [SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization](https://mlsanity.com/paper/2602.04811/) (2026)
- [DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories](https://mlsanity.com/paper/2602.10809/) (2026)
- [MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome](https://mlsanity.com/paper/2603.28407/) (2026)
- [Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?](https://mlsanity.com/paper/2604.03016/) (2026)
- [InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search](https://mlsanity.com/paper/2605.07510/) (2026)
- [S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agents](https://mlsanity.com/paper/2606.15367/) (2026)
- [InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents](https://mlsanity.com/paper/2510.02271/) (2025)
- [GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Traces](https://mlsanity.com/paper/2604.04017/) (2026)
- [MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments](https://mlsanity.com/paper/2604.13418/) (2026)
- [Interactive Evaluation Requires a Design Science](https://mlsanity.com/paper/2605.17829/) (2026)
- [MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents](https://mlsanity.com/paper/2607.11818/) (2026)
- [Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness](https://mlsanity.com/paper/2607.19322/) (2026)
- [LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake](https://mlsanity.com/paper/2606.10460/) (2026)

## Abstract (arXiv)

AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However, existing benchmarks such as BrowseComp primarily focus on textual content, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 400 challenging, hand-crafted questions designed to evaluate multimodal retrieval and reasoning capabilities. Unlike prior work, MM-BrowseComp incorporates visual prompts and necessitates the extraction of key evidence from web images and videos to complete questions, rendering text-only approaches insufficient. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of 27 state-of-the-art models reveals that even leading models like GPT-5-High with tools achieve only 24.25% accuracy, highlighting the suboptimal multimodal browsing capabilities, establishing MM-BrowseComp as a rigorous new standard for the field.