# SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support

arXiv: https://arxiv.org/abs/2604.08618 · HTML: https://mlsanity.com/paper/2604.08618/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 20 of 20)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents](https://mlsanity.com/paper/2602.02474/) (2026)
- [Symbolic Learning Enables Self-Evolving Agents](https://mlsanity.com/paper/2406.18532/) (2024)
- [Automatic Root Cause Analysis via Large Language Models for Cloud Incidents](https://mlsanity.com/paper/2305.15778/) (2023)
- [RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models](https://mlsanity.com/paper/2310.16340/) (2023)
- [D-Bot: Database Diagnosis System using Large Language Models](https://mlsanity.com/paper/2312.01454/) (2023)
- [A Survey of AIOps for Failure Management in the Era of Large Language Models](https://mlsanity.com/paper/2406.11213/) (2024)
- [Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support](https://mlsanity.com/paper/2510.06674/) (2025)

## Built on by (most cited first, 30 of 48)

- [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://mlsanity.com/paper/2605.23904/) (2026)
- [From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws](https://mlsanity.com/paper/2606.06324/) (2026)
- [MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolution](https://mlsanity.com/paper/2607.05297/) (2026)
- [SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibe](https://mlsanity.com/paper/2607.03451/) (2026)
- [SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution](https://mlsanity.com/paper/2605.18401/) (2026)
- [SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories](https://mlsanity.com/paper/2606.01311/) (2026)
- [SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing](https://mlsanity.com/paper/2606.14239/) (2026)
- [Agent Skill Evaluation and Evolution: Frameworks and Benchmarks](https://mlsanity.com/paper/2606.11435/) (2026)
- [VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation](https://mlsanity.com/paper/2606.08091/) (2026)
- [SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History](https://mlsanity.com/paper/2606.08671/) (2026)
- [SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems](https://mlsanity.com/paper/2606.01314/) (2026)
- [Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries](https://mlsanity.com/paper/2607.10113/) (2026)
- [Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?](https://mlsanity.com/paper/2605.22148/) (2026)
- [Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Libraries](https://mlsanity.com/paper/2605.10990/) (2026)
- [Unsupervised Skill Discovery for Agentic Data Analysis](https://mlsanity.com/paper/2606.06416/) (2026)
- [SPEAR: Code-Augmented Agentic Prompt Optimization](https://mlsanity.com/paper/2605.26275/) (2026)
- [ESC-Skills: Discovering and Self-Evolving Skills for Emotional Support Conversations](https://mlsanity.com/paper/2605.27908/) (2026)
- [SkillBrew: Multi-Objective Curation of Skill Banks for LLM Agents](https://mlsanity.com/paper/2605.29440/) (2026)
- [Rethinking Self-Evolution: A Constrained Exploration-Exploitation Process for Mitigating Skill Overfitting](https://mlsanity.com/paper/2607.26643/) (2026)
- [Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories](https://mlsanity.com/paper/2608.10319/) (2026)
- [Better, Faster, Stronger: Programmatic Skill Learning Best Reduces Agent Cost](https://mlsanity.com/paper/2608.11338/) (2026)
- [A Wrong Turn Does Not Ruin the Journey: Deviation-Guided Skill Self-Evolution for LLM Agents](https://mlsanity.com/paper/2609.29154/) (2026)
- [Certified Long-Horizon Code Agent Evolution via Validation-Gated Skill Optimization](https://mlsanity.com/paper/2609.32990/) (2026)
- [SkillPoison: Progressive Skill Poisoning via Successful Experiences](https://mlsanity.com/paper/2610.07645/) (2026)
- [You Live More Than Once: Towards Hierarchical Skill Meta-Evolving](https://mlsanity.com/paper/2605.28390/) (2026)
- [Workflow-to-Skill: Skill Creation via Routing-Workflow-Semantics-Attachments Decomposition](https://mlsanity.com/paper/2606.06893/) (2026)
- [SkillChain: Closing the Loop on Skill Evolution for Image-Based E-Commerce AI Assistants](https://mlsanity.com/paper/2606.12984/) (2026)
- [IDP AutoOpt: Agent-Driven Optimization of Document Processing Pipeline Configurations](https://mlsanity.com/paper/2607.26075/) (2026)
- [Self-Supervised Skill Optimization](https://mlsanity.com/paper/2607.28777/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)

## Abstract (arXiv)

Deploying LLM-powered agents in enterprise scenarios such as cloud technical support demands high-quality, domain-specific skills. However, existing skill creators lack domain grounding, producing skills poorly aligned with real-world task requirements. Moreover, once deployed, there is no systematic mechanism to trace execution failures back to skill deficiencies and drive targeted refinements, leaving skill quality stagnant despite accumulating operational evidence. We introduce SkillForge, a self-evolving framework that closes an end-to-end creation-evaluation-refinement loop. To produce well-aligned initial skills, a Domain-Contextualized Skill Creator grounds skill synthesis in knowledge bases and historical support tickets. To enable continuous self-optimization, a three-stage pipeline -- Failure Analyzer, Skill Diagnostician, and Skill Optimizer -- automatically diagnoses execution failures in batch, pinpoints the underlying skill deficiencies, and rewrites the skill to eliminate them. This cycle runs iteratively, allowing skills to self-improve with every round of deployment feedback. Evaluated on five real-world cloud support scenarios spanning 1,883 tickets and 3,737 tasks, experiments show that: (1) the Domain-Contextualized Skill Creator produces substantially better initial skills than the generic skill creator, as measured by consistency with expert-authored reference responses from historical tickets; and (2) the self-evolution loop progressively improves skill quality from diverse starting points (including expert-authored, domain-created, and generic skills) across successive rounds, demonstrating that automated evolution can surpass manually curated expert knowledge.