mlsanityData to 7 Oct 2026

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

arXiv 2607.28076 ↗First posted 30 Jul 2026cs.AI cs.LGcited by 4

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents. However, terminal rewards provide only coarse trajectory-level supervision, leaving successful behaviors, recurring mistakes, and incidental choices entangled in the same outcome signal. Existing agentic self-distillation methods enrich sparse supervision with natural-language skills, but skills retrieved externally or extracted from a single trajectory by stronger models may mismatch current experience, exceed the policy's capability, or remain path-specific. We propose Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts. For each prompt, the policy reflects on each verified trajectory in an on-policy group, and a stop-gradient snapshot contrasts the resulting reflections from successful and failed rollouts to construct group-level privileged guidance. Conditioned on this guidance, a self-teacher refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction. Experiments across multiple agentic environments and model scales demonstrate that GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 36 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

27 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning25 Jun
  2. 2026Self-Distilled Agentic Reinforcement Learning14 May
  3. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents12 Apr
  4. 2026Hindsight Credit Assignment for Long-Horizon LLM Agents7 Mar
  5. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
10 more
  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  3. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  4. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  5. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  6. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoningcited by 29
  7. 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoningcited by 11
  8. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  9. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
  10. 2026ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RLcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddingscited by 0
  2. 2026On-Policy Visual Evidence Distillationcited by 0
  3. 2026EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Makingcited by 0
  4. 2026My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic Reinforcement Learning with Self-Distilled Reward Shaping
  2. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
  3. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
  4. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  5. 2026PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
  6. 2026Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
  7. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
  8. 2026From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL