mlsanityData to 14 Oct 2026

Retrospective Progress-Aware Self-Refinement for LLM Agent Training

arXiv 2606.14302 ↗First posted 12 Jun 2026cs.CLcited by 6

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Long-horizon LLM-based agents receive rich environmental observations during interaction, yet outcome rewards provide limited explicit supervision about how individual actions advance task completion. We investigate whether agents can turn this interaction evidence into useful training signals through retrospective progress assessment. A WebShop pilot shows that direct progress prompting reduces task success, whereas hindsight-annotated demonstrations improve it. We introduce RePro, Retrospective Progress-Aware Training, with a forward-then-reflect rollout: the agent estimates progress while acting, then reassesses each step using the completed trajectory and outcome. After warmup with externally generated demonstrations, policy optimization combines self-generated progress differences, online-retrospective alignment, and format rewards with environment feedback, requiring neither a separate process reward model nor ongoing teacher annotation. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to 11.57% success rate gains.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 57 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

45 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agents20 Mar
  2. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs18 Dec
  3. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  4. 2025Agentic Entropy-Balanced Policy Optimization16 Oct
  5. 2025PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness27 Sep
  6. 2025Agentic Reinforced Policy Optimization26 Jul
  7. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  8. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  9. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  10. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  9. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  10. 2025Group Sequence Policy Optimizationcited by 835
  11. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  12. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
22 more
  1. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  2. 2024Token-Budget-Aware LLM Reasoningcited by 247
  3. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  4. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  5. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  6. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 64
  7. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  8. 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoningcited by 48
  9. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  10. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  11. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 33
  12. 2025ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflectioncited by 31
  13. 2025Self-Improving LLM Agents at Test-Timecited by 30
  14. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
  15. 2025SAMULE: Self-Learning Agents Enhanced by Multi-level Reflectioncited by 15
  16. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interactioncited by 15
  17. 2026Experiential Reflective Learning for Self-Improving LLM Agentscited by 13
  18. 2025ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflectioncited by 12
  19. 2026Adaptive Milestone Reward for GUI Agentscited by 12
  20. 2025PGPO: Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimizationcited by 8
  21. 2025SERL: Self-Examining Reinforcement Learning on Open-Domaincited by 5
  22. 2026PreFlect: From Retrospective to Prospective Reflection in Large Language Model Agentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 35
  2. 2026MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environmentscited by 1
  3. 2026The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?cited by 1
  4. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Modelscited by 0
  5. 2026Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Modelscited by 0
  6. 2026My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
  2. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
  3. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
  4. 2026Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards
  5. 2026No Time Like the Present: Agentic Test-Time Training for LLM Agents
  6. 2026MASkills: Continual Skills Optimization for Multi-Agent LLM Systems
  7. 2026DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
  8. 2026AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL