# Mining Agent Skills from Production Traces

arXiv: https://arxiv.org/abs/2610.05777 · HTML: https://mlsanity.com/paper/2610.05777/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 34 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 30 of 33)

- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning](https://mlsanity.com/paper/2602.08234/) (2026)
- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://mlsanity.com/paper/2605.23904/) (2026)
- [Inducing Programmatic Skills for Agentic Tasks](https://mlsanity.com/paper/2504.06821/) (2025)
- [Memp: Exploring Agent Procedural Memory](https://mlsanity.com/paper/2508.06433/) (2025)
- [How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior](https://mlsanity.com/paper/2505.16067/) (2025)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [LLMs4OL: Large Language Models for Ontology Learning](https://mlsanity.com/paper/2307.16648/) (2023)
- [SkillX: Automatically Constructing Skill Knowledge Bases for Agents](https://mlsanity.com/paper/2604.04804/) (2026)
- [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://mlsanity.com/paper/2512.10696/) (2025)
- [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026)
- [SkillNet: Create, Evaluate, and Connect AI Skills](https://mlsanity.com/paper/2603.04448/) (2026)
- [SkillGen: Verified Inference-Time Agent Skill Synthesis](https://mlsanity.com/paper/2605.10999/) (2026)
- [SkillFlow: Scalable and Efficient Agent Skill Retrieval System](https://mlsanity.com/paper/2504.06188/) (2025)
- [From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills](https://mlsanity.com/paper/2605.23899/) (2026)
- [SkillEvolver: Skill Learning as a Meta-Skill](https://mlsanity.com/paper/2605.10500/) (2026)
- [End-to-End Ontology Learning with Large Language Models](https://mlsanity.com/paper/2410.23584/) (2024)
- [More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries](https://mlsanity.com/paper/2605.24050/) (2026)
- [Not All Skills Help: Measuring and Repairing Agent Knowledge](https://mlsanity.com/paper/2606.15390/) (2026)
- [Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents](https://mlsanity.com/paper/2608.11888/) (2026)
- [SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills](https://mlsanity.com/paper/2605.24117/) (2026)
- [The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents](https://mlsanity.com/paper/2607.22520/) (2026)
- [Demystifying Agent Skills: Why They Work-Until They Don't](https://mlsanity.com/paper/2608.14036/) (2026)
- [One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows](https://mlsanity.com/paper/2608.19741/) (2026)

## Abstract (arXiv)

Agent skills that record procedural instructions are increasingly mined from execution traces rather than curated by hand. Skill-mining pipelines often use known task outcomes or feedback to guide skill construction. In production, reliable information on whether a run has succeeded may be unavailable. We study how the sampling of execution traces, access to success or failure information, and the form of the mined skills affect downstream task performance. Holding the mining pipeline fixed, we compare six combinations of mining evidence and skill forms. Mining evidence has three levels: successful trajectories only, successes and failures with their outcome labels, or the same mix with labels withheld. Skill form has two types: an ordered workflow plan, or a declarative ontology of entities, states, and policies. We evaluate the mined skills on two enterprise benchmarks, ThinkingBox-Bench and APEX-Agents. Analysis of task-level paired differences shows that the benefits of different configurations of mining evidence and skill forms depend on the enterprise domain. On ThinkingBox-Bench, paired differences show that workflows score better than ontology by 1.7 pp, Goldilocks beats success-only evidence type by 2.4 pp and Goldilocks blind simulating skills learnt without outcomes is worse by 3.1 pp. APEX-Agents shows a moderate preference for ontologies and no clear preference between evidence regimes. Within each domain, task structure related constraints drive uneven performance with mined skills. These findings motivate tailoring meta-skills to the demands of the target tasks rather than adopting a one-size-fits-all approach.