# Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution

arXiv: https://arxiv.org/abs/2610.18927 · HTML: https://mlsanity.com/paper/2610.18927/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 33)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [SoK: Agentic Skills -- Beyond Tool Use in LLM Agents](https://mlsanity.com/paper/2602.20867/) (2026)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills](https://mlsanity.com/paper/2605.23899/) (2026)
- [MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills](https://mlsanity.com/paper/2606.07131/) (2026)
- [Tool Preferences in Agentic LLMs are Unreliable](https://mlsanity.com/paper/2505.18135/) (2025)
- [HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?](https://mlsanity.com/paper/2604.15415/) (2026)
- [Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware](https://mlsanity.com/paper/2607.02357/) (2026)
- [Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures](https://mlsanity.com/paper/2607.28802/) (2026)
- [Agent Skill Evaluation and Evolution: Frameworks and Benchmarks](https://mlsanity.com/paper/2606.11435/) (2026)
- [How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations](https://mlsanity.com/paper/2512.07497/) (2025)
- [Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance](https://mlsanity.com/paper/2603.13404/) (2026)
- [Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis](https://mlsanity.com/paper/2605.00314/) (2026)
- [Aligning Requirement for Large Language Model's Code Generation](https://mlsanity.com/paper/2509.01313/) (2025)
- [Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents](https://mlsanity.com/paper/2608.11888/) (2026)
- [The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents](https://mlsanity.com/paper/2607.22520/) (2026)
- [ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox](https://mlsanity.com/paper/2605.10787/) (2026)
- [GuardianAgentBench: Where Agents Fail and How to Guard Them](https://mlsanity.com/paper/2607.20982/) (2026)
- [CASCADE: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation](https://mlsanity.com/paper/2604.19400/) (2026)
- [Do Skill Descriptions Tell the Truth? Detecting Undisclosed Security Behaviors in Code-Backed LLM Skills](https://mlsanity.com/paper/2605.12875/) (2026)
- [Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications](https://mlsanity.com/paper/2606.04769/) (2026)

## Abstract (arXiv)

LLM agent frameworks heavily rely on external skill assets to standardize tool usage, encode operational workflows, and supply executable runtime resources. Despite their widespread adoption, multi-component skill packages frequently suffer from cross-artifact inconsistencies, where documentation, scripts, and configuration files specify conflicting behavioral constraints. Although static auditing techniques can uncover such contractual discrepancies, static defect volume cannot reliably reflect real-world agent performance degradation. This gap leaves practitioners without rigorous evidence to assess the practical risk of flawed agent skills. This paper presents a systematic empirical study to demystify the runtime implications of skill inconsistencies. We conduct large-scale auditing over 575 community and benchmark skills, yielding 1,182 manually validated contract inconsistencies. We further build a controlled evaluation pipeline with 49 executable agent tasks and three representative LLMs across NoSkill, OriginalSkill, and FixedSkill experimental settings. Through fine-grained behavioral analysis of 441 paired execution trajectories, we quantify the actual performance impacts of resolved defects.