# AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery

arXiv: https://arxiv.org/abs/2604.25256 · HTML: https://mlsanity.com/paper/2604.25256/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 29 of 29)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025)
- [Language agents achieve superhuman synthesis of scientific knowledge](https://mlsanity.com/paper/2409.13740/) (2024)
- [AI-Researcher: Autonomous Scientific Innovation](https://mlsanity.com/paper/2505.18705/) (2025)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [WideSearch: Benchmarking Agentic Broad Info-Seeking](https://mlsanity.com/paper/2508.07999/) (2025)
- [PaSa: An LLM Agent for Comprehensive Academic Paper Search](https://mlsanity.com/paper/2501.10120/) (2025)
- [DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively](https://mlsanity.com/paper/2509.26603/) (2025)
- [LitSearch: A Retrieval Benchmark for Scientific Literature Search](https://mlsanity.com/paper/2407.18940/) (2024)
- [Toward Autonomous Long-Horizon Engineering for ML Research](https://mlsanity.com/paper/2604.13018/) (2026)
- [DeepCode: Open Agentic Coding](https://mlsanity.com/paper/2512.07921/) (2025)
- [PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature](https://mlsanity.com/paper/2510.10909/) (2025)
- [InfoFlow: Reinforcing Search Agent Via Reward Density Optimization](https://mlsanity.com/paper/2510.26575/) (2025)
- [Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives](https://mlsanity.com/paper/2601.20833/) (2026)
- [SAGE: Benchmarking and Improving Retrieval for Deep Research Agents](https://mlsanity.com/paper/2602.05975/) (2026)
- [GISA: A Benchmark for General Information-Seeking Assistant](https://mlsanity.com/paper/2602.08543/) (2026)
- [RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension](https://mlsanity.com/paper/2601.14289/) (2026)

## Built on by (most cited first, 11 of 11)

- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [AutoTrainess: Teaching Language Models to Improve Language Models Autonomously](https://mlsanity.com/paper/2606.31551/) (2026)
- [Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems](https://mlsanity.com/paper/2605.04018/) (2026)
- [Towards Recursive Self-Evolving Agentic Literature Retrieval](https://mlsanity.com/paper/2605.14306/) (2026)
- [Socratic agents for autonomous scientific discovery in high-dimensional physical systems](https://mlsanity.com/paper/2606.26722/) (2026)
- [DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation](https://mlsanity.com/paper/2605.21482/) (2026)
- [ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs](https://mlsanity.com/paper/2608.12788/) (2026)
- [WANDR: A Benchmark for Wide and Deep Research](https://mlsanity.com/paper/2608.14747/) (2026)
- [SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?](https://mlsanity.com/paper/2609.09113/) (2026)
- [Inspire: Benchmarking Scientific Literature Search for Open Research Problems](https://mlsanity.com/paper/2609.33233/) (2026)
- [Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts](https://mlsanity.com/paper/2609.38670/) (2026)

## Abstract (arXiv)

Autonomous scientific research is significantly advanced thanks to the development of AI agents. One key step in this process is finding the right scientific literature, whether to explore existing knowledge for a research problem, or to acquire evidence for verifying assumptions and supporting claims. To assess AI agents' capability in driving this process, we present AutoResearchBench, a dedicated benchmark for autonomous scientific literature discovery. AutoResearchBench consists of two complementary task types: (1) Deep Research, which requires tracking down a specific target paper through a progressive, multi-step probing process, and (2) Wide Research, which requires comprehensively collecting a set of papers satisfying given conditions. Compared to previous benchmarks on agentic web browsing, AutoResearchBench is distinguished along three dimensions: it is research-oriented, calling for in-depth comprehension of scientific concepts; literature-focused, demanding fine-grained utilization of detailed information; and open-ended, involving an unknown number of qualified papers and thus requiring deliberate reasoning and search throughout. These properties make AutoResearchBench uniquely suited for evaluating autonomous research capabilities, and extraordinarily challenging. Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many other strong baselines fall below 5%. We publicly release the dataset and evaluation pipeline to facilitate future research in this direction. We publicly release the dataset, evaluation pipeline, and code at https://github.com/CherYou/AutoResearchBench.