# Counterfactual Trace Auditing of LLM Agent Skills arXiv: https://arxiv.org/abs/2605.11946 ยท HTML: https://mlsanity.com/paper/2605.11946/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use. - Builds on 26 of its 29 references that are in the map. - First posted on arXiv 12 May 2026. ## Placed in - LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/) ## Builds on (most cited first, 26 of 26) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023) - [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023) - [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021) - [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021) - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023) - [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026) - [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026) - [Panoptic Scene Graph Generation with Semantics-Prototype Learning](https://mlsanity.com/paper/2307.15567/) (2023) - [Treble Counterfactual VLMs: A Causal Approach to Hallucination](https://mlsanity.com/paper/2503.06169/) (2025) - [Understanding In-Context Learning from Repetitions](https://mlsanity.com/paper/2310.00297/) (2023) - [DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detection](https://mlsanity.com/paper/2411.08227/) (2024) - [When Agents go Astray: Course-Correcting SWE Agents with PRMs](https://mlsanity.com/paper/2509.02360/) (2025) - [Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents](https://mlsanity.com/paper/2510.02837/) (2025) - [Secure On-Device Video OOD Detection Without Backpropagation](https://mlsanity.com/paper/2503.06166/) (2025) - [Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios](https://mlsanity.com/paper/2503.12374/) (2025) - [Mitigating Copy Bias in In-Context Learning through Neuron Pruning](https://mlsanity.com/paper/2410.01288/) (2024) - [Defenses Against Prompt Attacks Learn Surface Heuristics](https://mlsanity.com/paper/2601.07185/) (2026) - [Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure](https://mlsanity.com/paper/2604.02547/) (2026) - [SkillTester: Benchmarking Utility and Security of Agent Skills](https://mlsanity.com/paper/2603.28815/) (2026) - [The Autonomy Tax: Defense Training Breaks LLM Agents](https://mlsanity.com/paper/2603.19423/) (2026) ## Built on by (most cited first, 10 of 10) - [SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use](https://mlsanity.com/paper/2607.01874/) (2026) - [Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds](https://mlsanity.com/paper/2608.02636/) (2026) - [Auditing Harness Tampering in Self-Improving Agents](https://mlsanity.com/paper/2609.00069/) (2026) - [CatchBench: When Can an Agent Failure Be Caught?](https://mlsanity.com/paper/2608.22808/) (2026) - [Harnessing Agent Skills: Architectural Patterns and a Reference Architecture for Skill-Mediated LLM Agents](https://mlsanity.com/paper/2606.20631/) (2026) - [WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks](https://mlsanity.com/paper/2608.03499/) (2026) - [Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults](https://mlsanity.com/paper/2606.00914/) (2026) - [HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards](https://mlsanity.com/paper/2608.06012/) (2026) - [What Happens During Autonomous Deep Research After the User Steps Away?](https://mlsanity.com/paper/2609.33509/) (2026) - [Beyond Instruction Following: Learning Grounded Skill-Following with Skill Contracts](https://mlsanity.com/paper/2610.05161/) (2026) ## Abstract (arXiv) Large Language Model agents are increasingly augmented with agent skills. Current evaluation methods for skills remain limited. Most deployed benchmarks report only pass rate before and after a skill is attached, treating the skill as a black box change to agent behavior. We introduce Counterfactual Trace Auditing (CTA), a framework for measuring how a skill changes agent behavior. CTA pairs each with skill agent trace with a without skill counterpart on the same task, segments both traces into goal directed phases, aligns the phases, and emits structured Skill Influence Pattern (SIP) annotations. These annotations describe the behavioral effect of a skill rather than only its task outcome. We instantiate CTA on SWE-Skills-Bench with Claude across 49 software engineering tasks. The resulting audit reveals a clear evaluation gap. Pass rate changes by only +0.3 percentage points on average, suggesting little aggregate effect. Yet CTA identifies 522 SIP instances across the same paired traces, showing that the skills substantially reshape agent behavior even when pass rate is nearly unchanged. The audit also separates several recurring effects that pass rate cannot detect, including literal template copying, off task artifact creation, excess planning, and task recovery. Three findings emerge. First, high baseline tasks contain most of the observed skill effects, although their pass rate is already saturated and therefore cannot reflect those effects. Second, tasks with moderate baseline performance show the most recoverable gain, but often at substantially higher token cost. Third, the dominant SIP type can be identified by baseline bucket: surface anchoring is most common on ceiling tasks and edge-case prompting is most common on mid-range and floor tasks. These regularities turn informal failure mode observations into reproducible behavioral measurements.