# Rules to Tools: Executable Checks for LLM Agents in Scientific Computing

arXiv: https://arxiv.org/abs/2610.00313 · HTML: https://mlsanity.com/paper/2610.00313/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [CodePDE: An Inference Framework for LLM-driven PDE Solver Generation](https://mlsanity.com/paper/2505.08783/) (2025)
- [Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents](https://mlsanity.com/paper/2602.07900/) (2026)
- [Lang-PINN: From Language to Physics-Informed Neural Networks via a Multi-Agent Framework](https://mlsanity.com/paper/2510.05158/) (2025)
- [Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes](https://mlsanity.com/paper/2605.05724/) (2026)
- [PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation](https://mlsanity.com/paper/2605.09636/) (2026)
- [Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer](https://mlsanity.com/paper/2607.17100/) (2026)
- [The Implementation Lottery: Auditing Idea Reliability in Automated Research](https://mlsanity.com/paper/2607.26587/) (2026)
- [CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation](https://mlsanity.com/paper/2604.12268/) (2026)
- [scicode-lint: Detecting Methodology Bugs in Scientific Python Code with LLM-Generated Patterns](https://mlsanity.com/paper/2603.17893/) (2026)
- [SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models](https://mlsanity.com/paper/2608.04975/) (2026)
- [CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Development](https://mlsanity.com/paper/2607.26777/) (2026)
- [MDArena: Evaluating Coding Agents on Realistic Molecular Dynamics Workflows](https://mlsanity.com/paper/2608.02642/) (2026)
- [Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits](https://mlsanity.com/paper/2608.09740/) (2026)
- [LLM Assisted Coding with Metamorphic Specification Mutation Agent](https://mlsanity.com/paper/2511.18249/) (2025)
- [Teaching Code LLMs to Reason with Intermediate Formal Specifications](https://mlsanity.com/paper/2607.04232/) (2026)
- [Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents](https://mlsanity.com/paper/2608.16295/) (2026)

## Abstract (arXiv)

Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and budgets; the tool group receives a callable implementation. Across two task-ID cohorts, complete repair is 26/30 with text and 29/30 with the prepared checks. Three task IDs favor tools, one favors text, and eleven tie. The eight-ID cohort scores 13/16 versus 15/16, with a task-cluster bootstrap 95% interval of [-12.5, 43.75] percentage points for the difference. The larger shared-definition SciCode cohort ties at 13/24 per group. Five development-exposed tasks with alternate starting programs score 3/10 versus 7/10. The tool group favors tasks 17, 77, and 11; initial checks flag task 17 and report no violation for tasks 77 and 11. Task 37 favors text and has no initial reported violation. A fresh source-through-Python arm also reaches 15/16, matching the dedicated command's aggregate. In a matched PDE comparison, detailed text scores 23/24 and checks score 24/24, with 31.2% lower reported model output for checks. Agent-side output savings vary by cohort, while public CPU use rises in both task-ID cohorts. These results measure task-dependent repair outcomes and agent-side costs with prepared checks.