# CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

arXiv: https://arxiv.org/abs/2503.22708 · HTML: https://mlsanity.com/paper/2503.22708/

## Key facts

- Cited by 54 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 20 Mar 2025.

## Built on by (most cited first, 30 of 54)

- [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025)
- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents](https://mlsanity.com/paper/2510.07172/) (2025)
- [LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research](https://mlsanity.com/paper/2506.17335/) (2025)
- [AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise](https://mlsanity.com/paper/2507.00310/) (2025)
- [AI Scientists Fail Without Strong Implementation Capability](https://mlsanity.com/paper/2506.01372/) (2025)
- [RExBench: Can coding agents autonomously implement AI research extensions?](https://mlsanity.com/paper/2506.22598/) (2025)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Toward Autonomous Long-Horizon Engineering for ML Research](https://mlsanity.com/paper/2604.13018/) (2026)
- [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://mlsanity.com/paper/2604.13940/) (2026)
- [ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence](https://mlsanity.com/paper/2605.26340/) (2026)
- [DeepCode: Open Agentic Coding](https://mlsanity.com/paper/2512.07921/) (2025)
- [AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy](https://mlsanity.com/paper/2505.20538/) (2025)
- [ScholarEval: Research Idea Evaluation Grounded in Literature](https://mlsanity.com/paper/2510.16234/) (2025)
- [Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Science](https://mlsanity.com/paper/2506.04410/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery](https://mlsanity.com/paper/2605.23204/) (2026)
- [TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents](https://mlsanity.com/paper/2510.06579/) (2025)
- [Can AI agents conduct open-ended AI research? Early evidence from two case studies](https://mlsanity.com/paper/2607.27191/) (2026)
- [The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research](https://mlsanity.com/paper/2602.18458/) (2026)
- [MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility](https://mlsanity.com/paper/2605.16616/) (2026)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Learning to Predict Future-Aligned Research Proposals with Language Models](https://mlsanity.com/paper/2603.27146/) (2026)
- [FARS: A Fully Automated Research System Deployed at Scale](https://mlsanity.com/paper/2606.31651/) (2026)
- [The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learning](https://mlsanity.com/paper/2603.15914/) (2026)
- [Idea2Plan: Exploring AI-Powered Research Planning](https://mlsanity.com/paper/2510.24891/) (2025)
- [Generating Literature-Driven Scientific Theories at Scale](https://mlsanity.com/paper/2601.16282/) (2026)
- [OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving](https://mlsanity.com/paper/2604.21510/) (2026)
- [Recursive self-improvement of AI research agents](https://mlsanity.com/paper/2609.26457/) (2026)
- [The Ramon Llull's Thinking Machine for Automated Ideation](https://mlsanity.com/paper/2508.19200/) (2025)

## Abstract (arXiv)

Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e.g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code. In this work we introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain (like prompting a language model). We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments, with the system returning 19 discoveries, 6 of which were judged as being both at least minimally sound and incrementally novel after a multi-faceted evaluation beyond that typically conducted in prior work, including external (conference-style) review, code review, and replication attempts. Moreover, the discoveries span new tasks, agents, metrics, and data, suggesting a qualitative shift from benchmark optimization to broader discoveries.