# MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

arXiv: https://arxiv.org/abs/2603.28407 · HTML: https://mlsanity.com/paper/2603.28407/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 32)

- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines](https://mlsanity.com/paper/2409.12959/) (2024)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [WideSearch: Benchmarking Agentic Broad Info-Seeking](https://mlsanity.com/paper/2508.07999/) (2025)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research](https://mlsanity.com/paper/2505.19253/) (2025)
- [MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents](https://mlsanity.com/paper/2508.13186/) (2025)
- [OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data](https://mlsanity.com/paper/2603.15594/) (2026)
- [ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks](https://mlsanity.com/paper/2508.15804/) (2025)
- [LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild](https://mlsanity.com/paper/2510.14240/) (2025)
- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models](https://mlsanity.com/paper/2602.02185/) (2026)
- [ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry](https://mlsanity.com/paper/2507.16280/) (2025)
- [MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verification](https://mlsanity.com/paper/2603.15726/) (2026)
- [SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents](https://mlsanity.com/paper/2509.06283/) (2025)
- [Benchmark Test-Time Scaling of General LLM Agents](https://mlsanity.com/paper/2602.18998/) (2026)
- [DRBench: A Realistic Benchmark for Enterprise Deep Research](https://mlsanity.com/paper/2510.00172/) (2025)
- [MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents](https://mlsanity.com/paper/2601.12346/) (2026)
- [Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates](https://mlsanity.com/paper/2601.18510/) (2026)
- [Towards Personalized Deep Research: Benchmarks and Evaluations](https://mlsanity.com/paper/2509.25106/) (2025)
- [FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents](https://mlsanity.com/paper/2504.13128/) (2025)
- [MiroFlow: Towards High-Performance and Robust Open-Source Agent Framework for General Deep Research Tasks](https://mlsanity.com/paper/2602.22808/) (2026)
- [DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation](https://mlsanity.com/paper/2601.09688/) (2026)
- [DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation](https://mlsanity.com/paper/2512.17776/) (2025)
- [DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality](https://mlsanity.com/paper/2603.05912/) (2026)
- [GISA: A Benchmark for General Information-Seeking Assistant](https://mlsanity.com/paper/2602.08543/) (2026)
- [IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents](https://mlsanity.com/paper/2601.06676/) (2026)

## Built on by (most cited first, 6 of 6)

- [Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?](https://mlsanity.com/paper/2605.19196/) (2026)
- [Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness](https://mlsanity.com/paper/2607.19322/) (2026)
- [FinanceHarness: Autonomous Financial Deep Research Framework](https://mlsanity.com/paper/2607.27853/) (2026)
- [ViDR: Grounding Multimodal Deep Research Reports in Source Visual Evidence](https://mlsanity.com/paper/2605.13034/) (2026)
- [Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents](https://mlsanity.com/paper/2609.11243/) (2026)
- [NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities](https://mlsanity.com/paper/2609.15695/) (2026)

## Abstract (arXiv)

Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.