mlsanityData to 7 Oct 2026

From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training

arXiv 2607.16257 ↗First posted 28 Jun 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning (RL) has become a widely adopted technique for improving large language models (LLMs) on complex tasks. Despite this progress, existing RL methods still face challenges in training agents with longer-horizon interactions. One major bottleneck is distinguishing the contribution of different actions in long-horizon interaction, leading to high optimization variance. To address this, we introduce a novel policy gradient method, Hindsight Policy Optimization (HPO), that projects both the current policy distribution and the hindsight distribution into an intent space and extracts low-variance learning signals from the Wasserstein distance between them. We theoretically and empirically show that aggregating semantically similar states and actions in the intent space yields a bounded-variance estimator and improves policy performance stably. Our code is available online.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 45 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

34 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning10 Sep
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep
  3. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  5. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agents23 May
  6. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
16 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  2. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  3. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  4. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  5. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  6. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  7. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  8. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  9. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  10. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  11. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  12. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  13. 2023LAVA: Data Valuation without Pre-Specified Learning Algorithmscited by 69
  14. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  15. 2020Dataset Dynamics via Gradient Flows in Probability Spacecited by 18
  16. 2025From Reasoning to Code: GRPO Optimization for Underrepresented Languagescited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting
  2. 2026Learning More from Less: Reinforcement Learning from Hindsight
  3. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  4. 2026Learning Agentic Policy from Action Guidance
  5. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
  6. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
  7. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers
  8. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models