mlsanityData to 14 Oct 2026

Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities

arXiv 2508.20324 ↗First posted 27 Aug 2025cs.CLcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement Learning has emerged as a dominant post-training approach to elicit agentic RAG behaviors such as search and planning from language models. Despite its success with larger models, applying RL to compact models (e.g., 0.5--1B parameters) presents unique challenges. The compact models exhibit poor initial performance, resulting in sparse rewards and unstable training. To overcome these difficulties, we propose Distillation-Guided Policy Optimization (DGPO), which employs cold-start initialization from teacher demonstrations and continuous teacher guidance during policy optimization. To understand how compact models preserve agentic behavior, we introduce Agentic RAG Capabilities (ARC), a fine-grained metric analyzing reasoning, search coordination, and response synthesis. Comprehensive experiments demonstrate that DGPO enables compact models to achieve sophisticated agentic search behaviors, even outperforming the larger teacher model in some cases. DGPO makes agentic RAG feasible in computing resource-constrained environments.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 44 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

30 of 44 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
18 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  8. 2016Sequence-Level Knowledge Distillationcited by 801
  9. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  10. 2025Small Language Models are the Future of Agentic AIcited by 270
  11. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  12. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  13. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  14. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 64
  15. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  16. 2024Adaptive Retrieval-Augmented Generation for Conversational Systemscited by 22
  17. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
  18. 2024Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Taskscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reinforcement Learning with Verifiable Rewards for Small Search Agentscited by 0
  2. 2026Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraintscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  2. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
  3. 2025Search Wisely: Mitigating Sub-optimal Agentic Searches By Reducing Uncertainty
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  5. 2026Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning
  6. 2026GRASP: GRanularity-Aware Search Policy for Agentic RAG
  7. 2026AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing
  8. 2026Rethinking Agentic Reinforcement Learning In Large Language Models