mlsanityData to 7 Oct 2026

What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents

arXiv 2605.19447 ↗First posted 19 May 2026cs.AIcited by 9

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 40 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

35 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  2. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  4. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
18 more
  1. 2025Process Reinforcement through Implicit Rewardscited by 478
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  4. 2015Policy Distillationcited by 414
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  6. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  7. 2026Reinforcement Learning via Self-Distillationcited by 285
  8. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  9. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  10. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  11. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  12. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  13. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  14. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 18
  15. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problemscited by 12
  16. 2025NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problemscited by 11
  17. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
  18. 2026Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  2. 2026Agentic Reinforcement Learning with Observation-Calibrated Self-Distillationcited by 6
  3. 2026When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agentscited by 2
  4. 2026EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agentscited by 1
  5. 2026Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learningcited by 1
  6. 2026SERL-SQL: Selective Hindsight Distillation for Text-to-SQL Reinforcement Agentic Learningcited by 0
  7. 2026Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agentscited by 0
  8. 2026UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agentscited by 0
  9. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL
  2. 2026HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation
  3. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
  4. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
  5. 2026ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks
  6. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
  7. 2026AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
  8. 2026StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learning