Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
LLM agent frameworks heavily rely on external skill assets to standardize tool usage, encode operational workflows, and supply executable runtime resources. Despite their widespread adoption, multi-component skill packages frequently suffer from cross-artifact inconsistencies, where documentation, scripts, and configuration files specify conflicting behavioral constraints. Although static auditing techniques can uncover such contractual discrepancies, static defect volume cannot reliably reflect real-world agent performance degradation. This gap leaves practitioners without rigorous evidence to assess the practical risk of flawed agent skills. This paper presents a systematic empirical study to demystify the runtime implications of skill inconsistencies. We conduct large-scale auditing over 575 community and benchmark skills, yielding 1,182 manually validated contract inconsistencies. We further build a controlled evaluation pipeline with 49 executable agent tasks and three representative LLMs across NoSkill, OriginalSkill, and FixedSkill experimental settings. Through fine-grained behavioral analysis of 441 paired execution trajectories, we quantify the actual performance impacts of resolved defects.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 44 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
33 of 44 references.
Closest ancestors · 1
Same direction, newest first
Other references · 32
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 315
- 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
- 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 130
- 2026From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skillscited by 22
20 more
- 2026MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skillscited by 19
- 2025Tool Preferences in Agentic LLMs are Unreliablecited by 18
- 2026HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?cited by 11
- 2026Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malwarecited by 10
- 2026Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failurescited by 10
- 2026Agent Skill Evaluation and Evolution: Frameworks and Benchmarkscited by 8
- 2025How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulationscited by 7
- 2026Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performancecited by 7
- 2026Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesiscited by 7
- 2025Aligning Requirement for Large Language Model's Code Generationcited by 6
- 2026Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agentscited by 6
- 2026The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agentscited by 5
- 2026ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandboxcited by 4
- 2026GuardianAgentBench: Where Agents Fail and How to Guard Themcited by 4
- 2026CASCADE: Detecting Inconsistencies between Code and Documentation with Automatic Test Generationcited by 3
- 2026Do Skill Descriptions Tell the Truth? Detecting Undisclosed Security Behaviors in Code-Backed LLM Skillscited by 3
- 2026Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implicationscited by 2
- 2026When Specifications Meet Reality: Uncovering API Inconsistencies in Ethereum Infrastructurecited by 1
- 2026SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignmentcited by 1
- 2026SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Toolscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance
- 2026Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents
- 2025More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agents
- 2026AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents
- 2026When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
- 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
- 2026SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision
- 2026Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer