mlsanityData to 7 Oct 2026

CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

arXiv 2503.22708 ↗First posted 20 Mar 2025cs.AI cs.CLcited by 54

Not on the map. Why?

Cited by 23 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e.g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code. In this work we introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain (like prompting a language model). We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments, with the system returning 19 discoveries, 6 of which were judged as being both at least minimally sound and incrementally novel after a multi-faceted evaluation beyond that typically conducted in prior work, including external (conference-style) review, code review, and replication attempts. Moreover, the discoveries span new tasks, agents, metrics, and data, suggesting a qualitative shift from benchmark optimization to broader discoveries.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Mar 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 31.

54 papers.

Sort:

By area:

  1. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  3. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 32
  4. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  5. 2025AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprisecited by 28
  6. 2025AI Scientists Fail Without Strong Implementation Capabilitycited by 20
  7. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  8. 2025How Far Are AI Scientists from Changing the World?cited by 15
  9. 2026Toward Autonomous Long-Horizon Engineering for ML Researchcited by 15
  10. 2026AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilotcited by 15
  11. 2026ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidencecited by 15
  12. 2025DeepCode: Open Agentic Codingcited by 14
38 more
  1. 2025AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomycited by 13
  2. 2025ScholarEval: Research Idea Evaluation Grounded in Literaturecited by 10
  3. 2025Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Sciencecited by 9
  4. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  5. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 9
  6. 2025TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agentscited by 8
  7. 2026Can AI agents conduct open-ended AI research? Early evidence from two case studiescited by 7
  8. 2026The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Researchcited by 6
  9. 2026MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibilitycited by 6
  10. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  11. 2026Learning to Predict Future-Aligned Research Proposals with Language Modelscited by 5
  12. 2026FARS: A Fully Automated Research System Deployed at Scalecited by 5
  13. 2026The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learningcited by 4
  14. 2025Idea2Plan: Exploring AI-Powered Research Planningcited by 3
  15. 2026Generating Literature-Driven Scientific Theories at Scalecited by 3
  16. 2026OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solvingcited by 3
  17. 2026Recursive self-improvement of AI research agentscited by 3
  18. 2025The Ramon Llull's Thinking Machine for Automated Ideationcited by 2
  19. 2025Enhancing Agentic Autonomous Scientific Discovery with Vision-Language Model Capabilitiescited by 2
  20. 2025MirrorMind: Empowering OmniScientist with the Expert Perspectives and Collective Knowledge of Human Scientistscited by 2
  21. 2026Federation over Text: Insight Sharing for Multi-Agent Reasoningcited by 2
  22. 2026Knows: Agent-Native Structured Research Representationscited by 2
  23. 2026EvoSci: A Bio-Inspired Multi-Agent Framework for the Evolution of Scientific Discoverycited by 2
  24. 2025Augmenting Research Ideation with Data: An Empirical Investigation in Social Sciencecited by 1
  25. 2025THE-Tree: Can Tracing Historical Evolution Enhance Scientific Verification and Reasoning?cited by 1
  26. 2025From AutoRecSys to AutoRecLab: A Call to Build, Evaluate, and Govern Autonomous Recommender-Systems Research Labscited by 1
  27. 2025Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agentscited by 1
  28. 2026SciNav: A General Agent Framework for Scientific Coding Taskscited by 1
  29. 2026Towards Autonomous and Auditable Medical Imaging Model Developmentcited by 1
  30. 2026Can Large Language Models Forecast What Researchers Study Next?cited by 1
  31. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  32. 2025CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agentscited by 0
  33. 2026EvoMU: Evolutionary Machine Unlearningcited by 0
  34. 2026Grounding LLMs in Scientific Discovery via Embodied Actionscited by 0
  35. 2026A Blueprint for Self-Evolving Coding Agents in Vehicle Aerodynamic Drag Predictioncited by 0
  36. 2026The Scientific Contribution Graph: Automated Literature-based Technological Roadmapping at Scalecited by 0
  37. 2026ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discoverycited by 0
  38. 2026Baikal: Structured Search for Deep Research over Data Lakescited by 0

50 most cited of 54 All 54 (JSON)