mlsanityData to 14 Oct 2026

Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses

arXiv 2609.36630 ↗First posted 29 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agent skill learning and evolution also fits: General LLM agent frameworks and benchmarks

Abstract (arXiv)

Modern agents increasingly rely on memories, tools, and execution logic, so their competence extends beyond model parameters. This shift exposes a limitation of conventional knowledge distillation, which asks how a student model imitates a teacher model. We define Agent Distillation as the persistent transfer of task-solving knowledge from a teacher agent to a student agent. Our study organizes the field by where transferred knowledge is retained: within the model, as artifacts, through the execution harness, or across substrates. This perspective separates transfer evidence from its outcome and clarifies how knowledge moves between agent components. We develop an evaluation framework that relates retention to causal contribution and deployed utility. Together, these contributions establish a foundation for the reliable, maintainable, and safe development of increasingly complex agentic systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use. Builds on 66 of its 146 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

66 of 146 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026SKILL-KD: Contrastive Skill Distillation for LLM Agents30 Jul
  2. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning7 May
  3. 2026ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation26 Apr
  4. 2026SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks22 Apr
  5. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks13 Feb
  6. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning9 Feb

Other references · 60

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  9. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  10. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  11. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  12. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
38 more
  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  2. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  3. 2024Understanding the planning of LLM agents: A surveycited by 421
  4. 2024Agent Workflow Memorycited by 383
  5. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  6. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  7. 2023Large Language Models as Tool Makerscited by 303
  8. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  10. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  11. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  12. 2025Survey on Evaluation of LLM-based Agentscited by 204
  13. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  14. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 142
  15. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 141
  16. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  17. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  18. 2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agentscited by 74
  19. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 63
  20. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflowscited by 57
  21. 2026Self-Distilled Agentic Reinforcement Learningcited by 55
  22. 2025A Comprehensive Survey on Knowledge Distillationcited by 51
  23. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 47
  24. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 43
  25. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 31
  26. 2024WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Modelscited by 30
  27. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 30
  28. 2024Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Modelscited by 27
  29. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  30. 2023Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankerscited by 19
  31. 2025Imitation Learning for Multi-turn LM Agents via On-policy Expert Correctionscited by 19
  32. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 17
  33. 2026StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learningcited by 16
  34. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 15
  35. 2025A2Flow: Automating Agentic Workflow Generation via Self-Adaptive Abstraction Operatorscited by 14
  36. 2026General Agent Evaluationcited by 13
  37. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 12
  38. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learningcited by 12

50 most cited of 60 All 60 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security
  2. 2026Harnessing Agent Skills: Architectural Patterns and a Reference Architecture for Skill-Mediated LLM Agents
  3. 2026Subagents vs Agent Skills: Executing Reusable Knowledge for Long-Horizon Agentic Tasks
  4. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving
  5. 2026The Agent Use of Agent Beings: Agent Cybernetics Is the Missing Science of Foundation Agents
  6. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
  7. 2026Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries
  8. 2026The Epistemics of Agent Memory: Measuring, and Governing, the Consolidation Decision in Long-Horizon LLM Agents