# AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

arXiv: https://arxiv.org/abs/2510.21652 · HTML: https://mlsanity.com/paper/2510.21652/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2025.

## Built on by (most cited first, 30 of 53)

- [MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI](https://mlsanity.com/paper/2605.08678/) (2026)
- [Environment Scaling for Interactive Agentic Experience Collection: A Survey](https://mlsanity.com/paper/2511.09586/) (2025)
- [General Agent Evaluation](https://mlsanity.com/paper/2602.22953/) (2026)
- [RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards](https://mlsanity.com/paper/2605.10899/) (2026)
- [Benchmarking AI Agents for Addressing Scientific Challenges Across Scales](https://mlsanity.com/paper/2606.12736/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation](https://mlsanity.com/paper/2605.28655/) (2026)
- [SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks](https://mlsanity.com/paper/2605.31433/) (2026)
- [SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?](https://mlsanity.com/paper/2608.19799/) (2026)
- [BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks](https://mlsanity.com/paper/2602.06221/) (2026)
- [The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learning](https://mlsanity.com/paper/2603.15914/) (2026)
- [Herculean: An Agentic Benchmark for Financial Intelligence](https://mlsanity.com/paper/2605.14355/) (2026)
- [ASI-Bench: At the Dawn of Artificial Superintelligence](https://mlsanity.com/paper/2608.17271/) (2026)
- [Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations](https://mlsanity.com/paper/2604.27093/) (2026)
- [AI CFD Scientist: Toward Open-Ended Computational Fluid Dynamics Discovery with Physics-Aware AI Agents](https://mlsanity.com/paper/2605.06607/) (2026)
- [Agentics 2.0: Logical Transduction Algebra for Agentic Data Workflows](https://mlsanity.com/paper/2603.04241/) (2026)
- [Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users](https://mlsanity.com/paper/2603.16120/) (2026)
- [MaD Physics: Evaluating information seeking under constraints in physical environments](https://mlsanity.com/paper/2605.10820/) (2026)
- [Towards Recursive Self-Evolving Agentic Literature Retrieval](https://mlsanity.com/paper/2605.14306/) (2026)
- [ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment](https://mlsanity.com/paper/2606.00644/) (2026)
- [K-Bench: measuring model performance on real scientific agent requests](https://mlsanity.com/paper/2608.21601/) (2026)
- [AlphaEval: Evaluating Agents in Production](https://mlsanity.com/paper/2604.12162/) (2026)
- [FrontierChallenge: Evaluating Scientific Workflow Completion](https://mlsanity.com/paper/2608.24979/) (2026)
- [TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents](https://mlsanity.com/paper/2609.05079/) (2026)
- [ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery](https://mlsanity.com/paper/2605.16902/) (2026)
- [Explain Like I'm 5 or Whatever I Choose: Evaluating the Potential for Interactive Complexity in Language Model Responses](https://mlsanity.com/paper/2606.06788/) (2026)
- [Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes](https://mlsanity.com/paper/2606.15887/) (2026)
- [How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?](https://mlsanity.com/paper/2606.26346/) (2026)
- [Toward Trustworthy Autonomous Science: A Two-Year Community Roadmap](https://mlsanity.com/paper/2607.12113/) (2026)
- [Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation](https://mlsanity.com/paper/2607.25891/) (2026)

## Abstract (arXiv)

AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose "deep research" systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.