# AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents

arXiv: https://arxiv.org/abs/2602.06855 · HTML: https://mlsanity.com/paper/2602.06855/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 50 of its 97 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 50)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Talk like a Graph: Encoding Graphs for Large Language Models](https://mlsanity.com/paper/2310.04560/) (2023)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024)
- [MLGym: A New Framework and Benchmark for Advancing AI Research Agents](https://mlsanity.com/paper/2502.14499/) (2025)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [The Art of Scaling Reinforcement Learning Compute for LLMs](https://mlsanity.com/paper/2510.13786/) (2025)
- [CWM: An Open-Weights LLM for Research on Code Generation with World Models](https://mlsanity.com/paper/2510.02387/) (2025)

## Built on by (most cited first, 30 of 38)

- [ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence](https://mlsanity.com/paper/2605.26340/) (2026)
- [MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI](https://mlsanity.com/paper/2605.08678/) (2026)
- [ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research](https://mlsanity.com/paper/2606.07591/) (2026)
- [AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?](https://mlsanity.com/paper/2606.05080/) (2026)
- [AIRA_2: Overcoming Bottlenecks in AI Research Agents](https://mlsanity.com/paper/2603.26499/) (2026)
- [SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?](https://mlsanity.com/paper/2605.30329/) (2026)
- [Recursive self-improvement of AI research agents](https://mlsanity.com/paper/2609.26457/) (2026)
- [AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery](https://mlsanity.com/paper/2605.23204/) (2026)
- [NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?](https://mlsanity.com/paper/2606.24530/) (2026)
- [Can AI agents conduct open-ended AI research? Early evidence from two case studies](https://mlsanity.com/paper/2607.27191/) (2026)
- [AI Research Preference Models](https://mlsanity.com/paper/2608.13940/) (2026)
- [Training AI Scientists to Replicate Research](https://mlsanity.com/paper/2608.13331/) (2026)
- [Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering](https://mlsanity.com/paper/2607.28568/) (2026)
- [Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design](https://mlsanity.com/paper/2605.15871/) (2026)
- [GEAR: Genetic AutoResearch for Agentic Code Evolution](https://mlsanity.com/paper/2605.13874/) (2026)
- [What Do Evolutionary Coding Agents Evolve?](https://mlsanity.com/paper/2605.20086/) (2026)
- [ResearchLoop: An Evidence-Gated Control Plane for AI-Assisted Research](https://mlsanity.com/paper/2605.28282/) (2026)
- [AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers](https://mlsanity.com/paper/2607.29626/) (2026)
- [ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment](https://mlsanity.com/paper/2606.00644/) (2026)
- [TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoning](https://mlsanity.com/paper/2607.14658/) (2026)
- [K-Bench: measuring model performance on real scientific agent requests](https://mlsanity.com/paper/2608.21601/) (2026)
- [SciDER: Scientific Data-centric End-to-end Researcher](https://mlsanity.com/paper/2603.01421/) (2026)
- [Structured Abductive-Deductive-Inductive Reasoning for LLMs via Algebraic Invariants](https://mlsanity.com/paper/2604.15727/) (2026)
- [ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review](https://mlsanity.com/paper/2605.02651/) (2026)
- [Muse Spark Safety & Preparedness Report](https://mlsanity.com/paper/2606.12429/) (2026)
- [BrainPilot: Automating Brain Discovery with Agentic Research](https://mlsanity.com/paper/2607.15079/) (2026)
- [Argus: A General-Purpose Agentic Reasoning Runtime for Long-Horizon Tasks](https://mlsanity.com/paper/2608.05144/) (2026)
- [TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents](https://mlsanity.com/paper/2609.05079/) (2026)
- [When AI Designs AI: Innovation or Imitation?](https://mlsanity.com/paper/2608.17471/) (2026)
- [SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?](https://mlsanity.com/paper/2609.09113/) (2026)

## Abstract (arXiv)

LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.