mlsanityData to 7 Oct 2026

StateM: Reaching 95.3% Raw Accuracy, or a 15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

arXiv 2608.15089 ↗First posted 15 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Long-horizon agents can fail even when their underlying models can solve the constituent steps. They may lose track of mutable state, fail to reactivate lessons from earlier executions, skip known procedures, or stop prematurely. We bet on harness scaling to improve the execution system around an agent without changing its model weights. We introduce StateM, an agent-native runtime that organizes execution around durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that agents and users can inspect together. On Terminal-Bench 2.1, StateM raises GPT-5.5 xhigh to 92.1%, versus 83.1% reference and GPT-5.6 Sol Ultra at 91.9%. The runbook transfers unchanged to GPT-5.6. With GPT-5.6 Sol xhigh, StateM reaches 95.3% raw accuracy across 445 trials and succeeds on all 89 tasks at least once. The frozen profile raises GPT-5.6 Luna from 76.7 to 85.4%, above the 84.9% Sol xhigh reference. Using the same runtime, runbook structure, and golden rules, less than 38 of adaptation raises DeepSeek-V4 Flash from 82.7 to 88.1% under standard timeouts and to 89.1% on an 88-task common core. Extending only the remaining latency-sensitive task matches the reported 88.8% GPT-5.6 Sol max result. Final-score API usage is about 15 versus 574.68 for the GPT reference; total DeepSeek expenditure is 52.22. On BusinessBench, family-specific runbooks built on development sets yield held-out gains of 0.55 macro and 1.34 micro points; two mechanism-matched families improve by 10.04 points. Concrete rules generalize when tasks share execution structure, while the control methodology applies broadly. StateM turns selected postmortem findings into persistent, executable preconditions and practices, making learned controls explicit and enforceable through stateful controls. Code at github.com/henryqin1997/statem.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 12 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

11 of 12 references.

References · 11

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  2. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  3. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  4. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  5. 2026Self-Harness: Harnesses That Improve Themselvescited by 79
  6. 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65
  7. 2024Agent AI with LangGraph: A Modular Framework for Enhancing Machine Translation Using Large Language Modelscited by 27
  8. 2026Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agentscited by 22
  9. 2026From Model Scaling to System Scaling: Scaling the Harness in Agentic AIcited by 8
  10. 2026Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptationcited by 7
  11. 2026In-Context Prompting Obsoletes Agent Orchestration for Procedural Taskscited by 2

Built on by

Not cited yet.