mlsanityData to 14 Oct 2026

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

arXiv 2508.13186 ↗First posted 14 Aug 2025cs.CL cs.AI cs.CVcited by 42

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However, existing benchmarks such as BrowseComp primarily focus on textual content, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 400 challenging, hand-crafted questions designed to evaluate multimodal retrieval and reasoning capabilities. Unlike prior work, MM-BrowseComp incorporates visual prompts and necessitates the extraction of key evidence from web images and videos to complete questions, rendering text-only approaches insufficient. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of 27 state-of-the-art models reveals that even leading models like GPT-5-High with tools achieve only 24.25% accuracy, highlighting the suboptimal multimodal browsing capabilities, establishing MM-BrowseComp as a rigorous new standard for the field.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 37 of its 57 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

37 of 57 references.

References · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
25 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  5. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  6. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  7. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  8. 2021InfographicVQAcited by 437
  9. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  10. 2024Measuring short-form factuality in large language modelscited by 373
  11. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  12. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  13. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  14. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
  15. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  16. 2025BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinesecited by 145
  17. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 131
  18. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  19. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  20. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  21. 2025Visual Planning: Let's Think Only with Imagescited by 85
  22. 2025ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agentscited by 66
  23. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 43
  24. 2025BEARCUBS: A benchmark for computer-using web agentscited by 31
  25. 2025DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 37.

42 papers.

Sort:

By area:

  1. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 75
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  3. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  4. 2026REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agentscited by 49
  5. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 26
  6. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  7. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  8. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 22
  9. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
  10. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 17
  11. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  12. 2026VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agentscited by 13
30 more
  1. 2025InteractComp: Evaluating Search Agents With Ambiguous Queriescited by 12
  2. 2026Towards Long-horizon Agentic Multimodal Searchcited by 11
  3. 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 11
  4. 2026BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agentscited by 8
  5. 2026Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoningcited by 8
  6. 2026SE-Bench: Benchmarking Self-Evolution with Knowledge Internalizationcited by 6
  7. 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
  8. 2026MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcomecited by 6
  9. 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?cited by 5
  10. 2026InterLV-Search: Benchmarking Interleaved Multimodal Agentic Searchcited by 5
  11. 2026S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agentscited by 5
  12. 2025InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agentscited by 3
  13. 2026GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Tracescited by 3
  14. 2026MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environmentscited by 3
  15. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  16. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agentscited by 3
  17. 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
  18. 2026LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lakecited by 1
  19. 2026UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseCompcited by 1
  20. 2026WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agentscited by 1
  21. 2026MARCA: A Checklist-Based Benchmark for Multilingual Web Searchcited by 0
  22. 2026Struct-Searcher: Agentic Structural Thinking Advances Multimodal Deep Information Seekingcited by 0
  23. 2026DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Taskscited by 0
  24. 2026VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrievalcited by 0
  25. 2026Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agentscited by 0
  26. 2026NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communitiescited by 0
  27. 2026V-Gym: Enhancing Agentic Visual Reasoning via Skill-Data Co-Evolutioncited by 0
  28. 2026HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agentscited by 0
  29. 2026VideoResearchAgent: Grounded Task Synthesis and Sim-to-Real RL for Open-Web Video Researchcited by 0
  30. 2026Neos: Learning to Interleave Search and Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents
  2. 2026Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search
  3. 2026MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval
  4. 2024WebQuest: A Benchmark for Multimodal QA on Web Page Sequences
  5. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
  6. 2026K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
  7. 2025BrowseMaster: Towards Scalable Web Browsing via Tool-Augmented Programmatic Agent Pair
  8. 2026DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents