mlsanityData to 14 Oct 2026

Structured Agent Distillation for Large Language Model

arXiv 2505.13820 ↗First posted 20 May 2025cs.LG cs.AI cs.CLcited by 13

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) exhibit strong capabilities as decision-making agents by interleaving reasoning and actions, as seen in ReAct-style frameworks. Yet, their practical deployment is constrained by high inference costs and large model sizes. We propose Structured Agent Distillation, a framework that compresses large LLM-based agents into smaller student models while preserving both reasoning fidelity and action consistency. Unlike standard token-level distillation, our method segments trajectories into [REASON] and [ACT] spans, applying segment-specific losses to align each component with the teacher's behavior. This structure-aware supervision enables compact agents to better replicate the teacher's decision process. Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines, achieving significant compression with minimal performance drop. Scaling and ablation results further highlight the importance of span-level alignment for efficient and deployable agents.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 69 of its 108 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

69 of 108 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar
  3. 2022A Generalist Agent12 May

Other references · 66

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  9. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
38 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2023LIMA: Less Is More for Alignmentcited by 1,100
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  5. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  6. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  9. 2018Behavioral Cloning from Observationcited by 492
  10. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  12. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
  13. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 271
  14. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  15. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 153
  16. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  17. 2018CurriculumNet: Weakly Supervised Learning from Large-Scale Web Imagescited by 126
  18. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  19. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 105
  20. 2020ConvBERT: Improving BERT with Span-based Dynamic Convolutioncited by 88
  21. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 71
  22. 2023UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye Viewcited by 43
  23. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  24. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 37
  25. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 36
  26. 2021Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Searchcited by 34
  27. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  28. 2022Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterizationcited by 32
  29. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 29
  30. 2024Exploring Token Pruning in Vision State Space Modelscited by 28
  31. 2020LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuningcited by 25
  32. 2024Numerical Pruning for Efficient Autoregressive Modelscited by 25
  33. 2023ThoughtSource: A central hub for large language model reasoning datacited by 23
  34. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidancecited by 22
  35. 2022Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolutioncited by 20
  36. 2024Pruning Foundation Models for High Accuracy without Retrainingcited by 20
  37. 2025Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuningcited by 18
  38. 2024Squat: Quant Small Language Models on the Edgecited by 16

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 30
  2. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  3. 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 13
  4. 2026When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Makingcited by 2
  5. 2025TSLA: A Task-Specific Learning Adaptation for Semantic Segmentation on Autonomous Vehicles Platformcited by 1
  6. 2026AgentBrew: Lifelong Knowledge Brewing from Strong Teachers to Weak LLM Agentscited by 1
  7. 2026Test-Time Deep Thinking to Explore Implicit Rulescited by 0
  8. 2026Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8Bcited by 0
  9. 2026Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamicscited by 0
  10. 2026Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verificationcited by 0
  11. 2026Harness-Aware Distillation for Small Language Model Agentscited by 0
  12. 2026Learning to Act with Task Progress: Distilling Small Agents from Compact Teacher Supervisioncited by 0
1 more
  1. 2026Heterogeneous Agentic Coevolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Distilling LLM Agent into Small Models with Retrieval and Code Tools
  2. 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs
  3. 2026ActionOPD: Distilling Decisions, Not Tokens
  4. 2024Sub-goal Distillation: A Method to Improve Small Language Agents
  5. 2026Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
  6. 2026Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesis
  7. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  8. 2026Reasoning Compression with Mixed-Policy Distillation