# Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

arXiv: https://arxiv.org/abs/2608.23653 · HTML: https://mlsanity.com/paper/2608.23653/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 19 of 19)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [FEABench: Evaluating Language Models on Multiphysics Reasoning Ability](https://mlsanity.com/paper/2504.06260/) (2025)
- [SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Models](https://mlsanity.com/paper/2506.15695/) (2025)
- [Text2Model: Generating dynamic chemical reactor models using large language models (LLMs)](https://mlsanity.com/paper/2503.17004/) (2025)
- [SimuAgent: An LLM-Based Simulink Modeling Assistant Enhanced with Reinforcement Learning](https://mlsanity.com/paper/2601.05187/) (2026)
- [Metadata practices for simulation workflows](https://mlsanity.com/paper/2408.17309/) (2024)
- [ModiGen: A Large Language Model-Based Workflow for Multi-Task Modelica Code Generation](https://mlsanity.com/paper/2503.18460/) (2025)
- [Coupled Requirements-driven Testing of CPS: From Simulation To Reality](https://mlsanity.com/paper/2403.16287/) (2024)
- [PowerAgentBench-SS: A Benchmark for Agentic AI in Power System Steady-State Studies](https://mlsanity.com/paper/2606.18789/) (2026)
- [PowerAgentBench-Dyn: A Benchmark for Agentic AI in Power System Dynamic Studies](https://mlsanity.com/paper/2606.20401/) (2026)
- [Automating Modelica Module Generation Using Large Language Models: A Case Study on Building Control Description Language](https://mlsanity.com/paper/2509.14623/) (2025)
- [Simulation Code Generation for Fluid Systems using Large Language Models: Benchmarking Models and Prompting Strategies](https://mlsanity.com/paper/2607.29389/) (2026)
- [ModBench: A Pipeline for Building Modelica Benchmark Datasets Mined from Library Repositories](https://mlsanity.com/paper/2608.16638/) (2026)

## Abstract (arXiv)

AI agents are increasingly used for simulation-driven engineering. Physical system modeling presents different requirements from general-purpose code generation in software engineering, because correctness depends not only on syntax and executability but also on physical consistency and scenario-dependent behavior. We study this challenge in Modelica, an equation-based modeling language in which a model may compile and simulate while still violating its intended physics or engineering requirements. Across successive revisions, an agent may lose track of requirements or rely on simulation evidence produced by an outdated candidate. To address this challenge, we present Pufibara, an agent harness that maintains persistent engineering state across revisions, associates execution and simulation evidence with the candidate that produced it, and makes submission an explicit agent action. To evaluate end-to-end Modelica agent workflows, we also propose a source-grounded method for constructing realistic and independently evaluable tasks. We use this method to build the 232-task Modelica Agent Workflow Benchmark, spanning Model Repair, Model Generation, and Model Tuning. Each submitted candidate is scored by a benchmark-owned evaluator outside the agent loop. We compare Pufibara with Claude Code as complete harnesses under two matched large language model (LLM) backends. With DeepSeek v4 Flash, Pufibara passes 202 tasks, compared with 185 for Claude Code. With Claude Sonnet 5, Pufibara passes 202 tasks, compared with 187 for Claude Code. Under the repository-reported token accounting, Pufibara records 76.4%-82.5% lower logical-token totals. Its sequential runtime is 6.1%-58.4% lower. These findings show that, even under matched LLM backends, complete agent harnesses can differ substantially in both task success and resource use for physical system modeling.