mlsanityData to 7 Oct 2026

HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning

arXiv 2608.01597 ↗First posted 3 Aug 2026cs.LG cs.AI cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge's access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

30 of 41 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
18 more
  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  3. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  4. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  5. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  6. 2025s3: You Don't Need That Much Data to Train a Search Agent via RLcited by 41
  7. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 34
  8. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learningcited by 32
  9. 2025Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agentcited by 28
  10. 2026Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Trainingcited by 15
  11. 2026AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabelingcited by 13
  12. 2026SmartSearch: Process Reward-Guided Query Refinement for Search Agentscited by 12
  13. 2025Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewritingcited by 10
  14. 2026Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoningcited by 9
  15. 2026IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoningcited by 8
  16. 2026HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agentscited by 8
  17. 2026TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimizationcited by 5
  18. 2026SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
  2. 2026Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
  3. 2026IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents
  4. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
  5. 2026From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training
  6. 2026Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
  7. 2026H2SD: Hybrid Hindsight Self-Distillation
  8. 2026Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating