# HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents

arXiv: https://arxiv.org/abs/2605.17873 · HTML: https://mlsanity.com/paper/2605.17873/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [OpenClaw-RL: Train Any Agent Simply by Talking](https://mlsanity.com/paper/2603.10165/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [AgentEvolver: Towards Efficient Self-Evolving Agent System](https://mlsanity.com/paper/2511.10395/) (2025)
- [Hindsight Credit Assignment for Long-Horizon LLM Agents](https://mlsanity.com/paper/2603.08754/) (2026)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost](https://mlsanity.com/paper/2603.21383/) (2026)

## Built on by (most cited first, 10 of 10)

- [Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.28424/) (2026)
- [When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents](https://mlsanity.com/paper/2608.05219/) (2026)
- [DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents](https://mlsanity.com/paper/2608.18524/) (2026)
- [Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.07371/) (2026)
- [Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses](https://mlsanity.com/paper/2609.36630/) (2026)
- [Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2609.36864/) (2026)
- [AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation](https://mlsanity.com/paper/2609.38142/) (2026)
- [Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation](https://mlsanity.com/paper/2610.02700/) (2026)
- [ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience](https://mlsanity.com/paper/2610.05303/) (2026)
- [Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents](https://mlsanity.com/paper/2610.19834/) (2026)

## Abstract (arXiv)

Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only to targeted action spans. Experiments on BFCL v3 and AppWorld show that our method outperforms the dense per-turn feedback baseline by up to 13.60 percentage points on average while achieving a 2.26× reduction in time per training step, suggesting that selecting where to distill is key to effective and efficient long-horizon agent training.