mlsanityData to 7 Oct 2026

Making Agents More Consistent: Skills Should Form Habits for Repeat Tasks

arXiv 2609.25299 ↗First posted 21 Sep 2026cs.AI cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

On repeated work, agents are inconsistent. We ran 42 tasks three times each and found that, depending on the model, 38% to 74% returned answers that did not agree. Consistency is what a buyer, an auditor, or a regulator requires, and agents do not have it. They are wasteful too: 95.3% to 97.2% of what an agent generates goes to re-deriving a plan the system already knows. We propose skill habit formation. An agent mines its own execution history for candidate skills, deterministic variants that compete against the incumbent rather than replacing it. A candidate declares the region of input space it claims, so the common case runs as a script and the rest falls through to reasoning. Four gates of ascending cost admit candidates; the central one tests a candidate's execution trace against a retained reference, within a tolerance measured from that reference's own run-to-run variability. On text-to-SQL, three of four reasoning arms reproduced their own output on 11 to 13 of 42 repeated questions and the fourth on 26 of 42, while a habit-formed variant reproduced on all 456 dispatches we repeated and was non-inferior to every arm it replaced (p<0.0001). It also used 14% to 56% fewer tokens, turning net positive after 7 to 53 reuses. We measured what this costs in accuracy. The guard admitted work it should have deferred on 2.6% of natural paraphrases and 26% of inputs near its boundary, and 11 of 13 such failures were invisible to the trace-conformance gate at any threshold. Deterministic errors repeat exactly: a bad habit is as reliable as a good one, and that is the price of the property that makes the system auditable. Separating routing from parameter extraction raised end-to-end accuracy from 0.888 to 0.952 at 43% of the cost.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 41 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

13 of 41 references.

References · 13

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2017Safe Reinforcement Learning via Shieldingcited by 427
  8. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  9. 2024Agent Workflow Memorycited by 342
  10. 2023An LLM Compiler for Parallel Function Callingcited by 170
  11. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 150
  12. 2015Shield Synthesis: Runtime Enforcement for Reactive Systemscited by 41
1 more
  1. 2026The Digital Apprentice: A Framework for Human-Directed Agentic AI Developmentcited by 1

Built on by

Not cited yet.