# Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model-Skill Co-Evolution

arXiv: https://arxiv.org/abs/2610.12367 · HTML: https://mlsanity.com/paper/2610.12367/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 30 of 32)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning](https://mlsanity.com/paper/2602.08234/) (2026)
- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [SkillOpt: Executive Strategy for Self-Evolving Agent Skills](https://mlsanity.com/paper/2605.23904/) (2026)
- [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://mlsanity.com/paper/2605.00674/) (2026)
- [CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification](https://mlsanity.com/paper/2604.01687/) (2026)
- [EvoSkill: Automated Skill Discovery for Multi-Agent Systems](https://mlsanity.com/paper/2603.02766/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?](https://mlsanity.com/paper/2603.15401/) (2026)
- [Mathematical exploration and discovery at scale](https://mlsanity.com/paper/2511.02864/) (2025)
- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce](https://mlsanity.com/paper/2506.06576/) (2025)
- [Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning](https://mlsanity.com/paper/2605.06130/) (2026)
- [Tmax: A simple recipe for terminal agents](https://mlsanity.com/paper/2606.23321/) (2026)
- [Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors](https://mlsanity.com/paper/2509.13237/) (2025)
- [SkillFlow: Scalable and Efficient Agent Skill Retrieval System](https://mlsanity.com/paper/2504.06188/) (2025)
- [WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://mlsanity.com/paper/2608.27454/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [Procedural Knowledge at Scale Improves Reasoning](https://mlsanity.com/paper/2604.01348/) (2026)
- [Co-Evolving Skill Generation and Policy Optimization](https://mlsanity.com/paper/2606.08755/) (2026)
- [Hybrid-Gym: Training Coding Agents to Generalize Across Tasks](https://mlsanity.com/paper/2602.16819/) (2026)
- [Skill-Conditioned Gated Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2605.28791/) (2026)

## Abstract (arXiv)

Skills, reusable procedural guidance added at inference, can substantially improve LLM downstream performance (Li et al., 2026). Prior work retrieves skills from a bank by semantic relevance, then uses them as inference-time patches or for model distillation. The individual utility of each skill, however, is largely neglected. We first show that, in on-policy distillation where skill-conditioned policies serve as teachers, fewer than 25% of retrieved skills provide useful distillation signals. We then propose SGUID, a method for selecting a compact subset of skills for distillation. SGUID retains a skill only if it consistently yields effective learning signals during training. The selected skills are then distilled to produce a better model. Our results show that not all skills are worth distilling. Across four models from the Olmo and Qwen families, distilling 6 selected skills matches or exceeds full-bank distillation in mean avg@12 on three of the four models, and on all four after a second round that distills 3 newly selected skills, while the full banks are up to 11x larger. Importantly, SGUID supports stable model-skill co-evolution: after a distillation round, a new candidate bank is curated from the updated model's rollouts, and SGUID selects which skills to internalize next. In the second round, this loop selects 3 new skills and improves Qwen3-8B from 64.3% to 66.3%. The selection step is essential for stability: on Qwen3-4B, naively updating the model with unfiltered skills degrades performance, including a 0.3 percentage point drop on HMMT25, whereas SGUID improves HMMT25 by 0.5 points after the first round and 1.1 points after the second. These results identify skill selection as the key mechanism for stable model-skill co-evolution.