Who Holds the Pen? Let Specifications, Not Agents, Sign Off
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks also fits: Agent skill learning and evolution
Abstract (arXiv)
Large language model agents increasingly combine generation, decision-making, execution, and self-evaluation within a single agentic loop. Although they operate under external specifications such as task instructions, guidelines, output schemas, and reusable skills, these specifications typically remain context for the same model that acts and declares completion, leaving no independent specification authority boundary. We identify two resulting gaps. The understanding--execution gap arises when a requirement is understood but not satisfied in execution; the state--authority gap arises when an agent's interpretation or completion claim does not establish the required state. On SkillsBench, using only agent-visible prompts, workspace information, and injected skill specifications, we extract 509 source-grounded task directions. Across seven models, only 79.6%--86.4% are satisfied, while completion-claim rates exceed official evaluator pass rates by 28.7--37.9 percentage points. We therefore separate agent proposals from authoritative state. Agents may plan, act, and request completion, but only admissible evidence from qualified providers may establish specification-governed state. SpecHarness operationalizes this principle by compiling visible specifications into source-linked obligations and governing execution and finalization through versioned obligation state. Verifiable requirements are mediated or validated at runtime, while ambiguous or subjective requirements remain advisory. Experiments on guideline-following and artifact-generation tasks show that specifications can serve not merely as behavioral guidance, but as authority over compliant execution and completion.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 15 of its 25 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
15 of 25 references.
Closest ancestors · 1
Same direction, newest first
Other references · 14
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
- 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
- 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
- 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
- 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectoriescited by 53
- 2024AgentOps: Enabling Observability of LLM Agentscited by 46
- 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 46
- 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 22
- 2025RvLLM: LLM Runtime Verification with Domain Knowledgecited by 15
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Agent Harnesses: Cross-Substrate Authority for Multi-Agent Systems
- 2026The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents
- 2026Designing for Doubt: The Case for Informed Abstention in Autonomous Agents
- 2026FORTIS: Benchmarking Over-Privilege in Agent Skills
- 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward
- 2026HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
- 2026Subagents vs Agent Skills: Executing Reusable Knowledge for Long-Horizon Agentic Tasks
- 2026From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills