mlsanityData to 7 Oct 2026

Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement

arXiv 2406.11176 ↗First posted 17 Jun 2024cs.CL cs.AI cs.LGcited by 86

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language model agents have exhibited exceptional performance across a range of complex interactive tasks. Recent approaches have utilized tuning with expert trajectories to enhance agent performance, yet they primarily concentrate on outcome rewards, which may lead to errors or suboptimal actions due to the absence of process supervision signals. In this paper, we introduce the Iterative step-level Process Refinement (IPR) framework, which provides detailed step-by-step guidance to enhance agent training. Specifically, we adopt the Monte Carlo method to estimate step-level rewards. During each iteration, the agent explores along the expert trajectory and generates new actions. These actions are then evaluated against the corresponding step of expert trajectory using step-level rewards. Such comparison helps identify discrepancies, yielding contrastive action pairs that serve as training data for the agent. Our experiments on three complex agent tasks demonstrate that our framework outperforms a variety of strong baselines. Moreover, our analytical findings highlight the effectiveness of IPR in augmenting action efficiency and its applicability to diverse models.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 27 of its 51 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

27 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan
  2. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk10 Jan
  3. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
  4. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  5. 2023FireAct: Toward Language Agent Fine-tuning9 Oct

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  12. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
10 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  3. 2023Large Language Model Alignment: A Surveycited by 241
  4. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  5. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  6. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  7. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  8. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  9. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  10. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 41; 2026: 37.

86 papers.

Sort:

By area:

  1. 2025Agentic Reinforced Policy Optimizationcited by 166
  2. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  3. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  4. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  5. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  6. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  7. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  8. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuningcited by 51
  9. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  10. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  11. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  12. 2025MPO: Boosting LLM Agents with Meta Plan Optimizationcited by 36
38 more
  1. 2025ATLaS: Agent Tuning via Learning Critical Stepscited by 29
  2. 2025AgentRM: Enhancing Agent Generalization with Reward Modelingcited by 25
  3. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
  4. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 22
  5. 2025Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learningcited by 22
  6. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  7. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planningcited by 19
  8. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 19
  9. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  10. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  11. 2025Learning Hierarchical Procedural Memory for LLM Agents through Bayesian Selection and Contrastive Refinementcited by 17
  12. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 17
  13. 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Usecited by 16
  14. 2025SAND: Boosting LLM Agents with Self-Taught Action Deliberationcited by 14
  15. 2026Verified Critical Step Optimization for LLM Agentscited by 13
  16. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  17. 2026Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflowscited by 11
  18. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 10
  19. 2025Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Gamecited by 10
  20. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  21. 2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Taskscited by 10
  22. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  23. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directionscited by 9
  24. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregationcited by 9
  25. 2025Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learningcited by 9
  26. 2025Exploring Expert Failures Improves LLM Agent Tuningcited by 7
  27. 2025Co-Evolving Agents: Learning from Failures as Hard Negativescited by 7
  28. 2026Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Modelscited by 7
  29. 2026PABU: Progress-Aware Belief Update for Efficient LLM Agentscited by 6
  30. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 6
  31. 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Trainingcited by 6
  32. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Taskscited by 6
  33. 2026Doctor-RAG: A Failure-Aware Repair Framework for Agentic Retrieval-Augmented Generationcited by 5
  34. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 5
  35. 2025Semantic Probabilistic Control of Language Modelscited by 4
  36. 2025Tru-POMDP: Task Planning Under Uncertainty via Tree of Hypotheses and Open-Ended POMDPscited by 4
  37. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agentscited by 4
  38. 2025DEPO: Dual-Efficiency Preference Optimization for LLM Agentscited by 4

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
  2. 2025Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking
  3. 2026SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
  4. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback
  5. 2024One STEP at a time: Language Agents are Stepwise Planners
  6. 2026Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills
  7. 2026Trace2Tower: Transition-Aware EigenTrace Induction of Multi-Level Skills for LLM Agents
  8. 2024Empowering Large Language Model Agents through Action Learning