# AInsteinBench: Benchmarking Coding Agents on Scientific Repositories

arXiv: https://arxiv.org/abs/2512.21373 · HTML: https://mlsanity.com/paper/2512.21373/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 24 Dec 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 16 of 16)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023)
- [OpenMM 8: Molecular Dynamics Simulation with Machine Learning Potentials](https://mlsanity.com/paper/2310.03121/) (2023)
- [How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM](https://mlsanity.com/paper/2504.05786/) (2025)
- [SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models](https://mlsanity.com/paper/2401.07950/) (2024)
- [Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation](https://mlsanity.com/paper/2407.08940/) (2024)
- [Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab](https://mlsanity.com/paper/2507.02083/) (2025)
- [Leveraging Large Language Models for Code Translation and Software Development in Scientific Computing](https://mlsanity.com/paper/2410.24119/) (2024)

## Built on by (most cited first, 6 of 6)

- [Immersion in the GitHub Universe: Scaling Coding Agents to Mastery](https://mlsanity.com/paper/2602.09892/) (2026)
- [MDGYM: Benchmarking AI Agents on Molecular Simulations](https://mlsanity.com/paper/2605.08941/) (2026)
- [ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments](https://mlsanity.com/paper/2609.19134/) (2026)
- [An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc](https://mlsanity.com/paper/2603.15976/) (2026)
- [AlphaEval: Evaluating Agents in Production](https://mlsanity.com/paper/2604.12162/) (2026)
- [Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks](https://mlsanity.com/paper/2609.06780/) (2026)

## Abstract (arXiv)

We introduce AInsteinBench, a large-scale benchmark for evaluating whether large language model (LLM) agents can operate as scientific computing development agents within real research software ecosystems. Unlike existing scientific reasoning benchmarks which focus on conceptual knowledge, or software engineering benchmarks that emphasize generic feature implementation and issue resolving, AInsteinBench evaluates models in end-to-end scientific development settings grounded in production-grade scientific repositories. The benchmark consists of tasks derived from maintainer-authored pull requests across six widely used scientific codebases, spanning quantum chemistry, quantum computing, molecular dynamics, numerical relativity, fluid dynamics, and cheminformatics. All benchmark tasks are carefully curated through multi-stage filtering and expert review to ensure scientific challenge, adequate test coverage, and well-calibrated difficulty. By leveraging evaluation in executable environments, scientifically meaningful failure modes, and test-driven verification, AInsteinBench measures a model's ability to move beyond surface-level code generation toward the core competencies required for computational scientific research.