Canonical Procedural Actions: An Auditable Annotation Protocol for Tool-Use Agent Traces
On the map: LLM agents and tool use → Software engineering coding agents also fits: Tool learning and function calling
Abstract (arXiv)
Tool-use agent traces identify messages and API calls, but procedural analyses also need explicit units of action and inspectable links to their evidence. We present Canonical Procedural Actions (CPAs), an annotation protocol that records a procedural function, its first agent-event anchor, the agent events that realize it, and separate contextual evidence. Multiple actions may share a message anchor without an inferred within-message order. A retail case study produces a versioned 24-entry codebook through open induction, recorded consolidation, and successive application audits. Two isolated LLM contexts annotate 32 trajectories disjoint from development at the trajectory level, producing 499 and 491 occurrences with anchor-label overlap A=0.982. Requiring identical context-event references reduces overlap to 0.798. These are structural repeatability measures, not semantic accuracy: 16 of 26 task IDs also occur in development, and historical tool payloads were truncated to 110 characters. Retrospective controls show that collapsing all labels raises overlap to 0.986, while simple endpoint rules reproduce the tool-anchored portion with 0.997 overlap. Assistant-message actions have 0.971 overlap, with a per-label minimum of 0.816. Applying the frozen codebook to 244 further trajectories yields 4,058 records, including eight diagnostic outcomes. The contribution is an explicit, auditable annotation instrument and a case study of its construction and measurement limits; human-reference validity and downstream utility remain to be established.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 6 of its 19 references that are in the map. First posted on arXiv 21 Sep 2026.
Builds on
6 of 19 references.
References · 6
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
- 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
- 2025Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Taskscited by 21
- 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 16
- 2026Agent trajectories as programs: fingerprinting and programming coding-agent behaviorcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflows
- 2025TRAIL: Trace Reasoning and Agentic Issue Localization
- 2026When Tool Outputs Become Commands: Separating Action Induction from Runtime Authorization in Tool-Augmented LLM Agents
- 2026TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces
- 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents
- 2026LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents
- 2026Projecting the Emerging Mindset of SWE Agent by Launching a Wild Code Understanding Journey
- 2026Agent Flight Recorder: Tamper-Evident Audit Trails with On-Chain Anchoring for Long-Horizon Tool-Using Agents