mlsanityData to 7 Oct 2026

Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents

arXiv 2403.02502 ↗First posted 4 Mar 2024cs.CL cs.AI cs.LGcited by 194

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large Language Models (LLMs) have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subsequent training phase, the agent utilizes these trajectory preference pairs to update its policy using contrastive learning methods like DPO. This iterative cycle of exploration and training fosters continued improvement in the agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin. Furthermore, an examination of task-solving efficiency and potential in scenarios lacking expert trajectory underscores the effectiveness of our approach.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 40 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

29 of 40 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Mistral 7Bcited by 3,065
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
17 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  8. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  9. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  11. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  12. 2023Large Language Model Alignment: A Surveycited by 241
  13. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  14. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  15. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  16. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  17. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 82; 2026: 77.

194 papers.

Sort:

By area:

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  2. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  3. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  4. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  5. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  6. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  7. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  8. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  9. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  10. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  11. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  12. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
38 more
  1. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  2. 2024Symbolic Learning Enables Self-Evolving Agentscited by 104
  3. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  4. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  5. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  6. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  7. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  8. 2024Beyond Browsing: API-Based Web Agentscited by 77
  9. 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectoriescited by 75
  10. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  11. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  12. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  13. 2024Agent Planning with World Knowledge Modelcited by 66
  14. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 66
  15. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 63
  16. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  17. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  18. 2026SkillNet: Create, Evaluate, and Connect AI Skillscited by 48
  19. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 45
  20. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  21. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  22. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  23. 2025Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agentscited by 39
  24. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  25. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  26. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learningcited by 37
  27. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  28. 2024Benchmarking Agentic Workflow Generationcited by 36
  29. 2025MPO: Boosting LLM Agents with Meta Plan Optimizationcited by 36
  30. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verificationcited by 35
  31. 2025A Survey of Direct Preference Optimizationcited by 35
  32. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  33. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  34. 2025Enabling Agents to Communicate Entirely in Latent Spacecited by 31
  35. 2024DynaSaur: Large Language Agents Beyond Predefined Actionscited by 30
  36. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  37. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Modelscited by 28
  38. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  2. 2025A Survey on the Optimization of Large Language Model-based Agents
  3. 2026Look Before You Leap: Autonomous Exploration for LLM Agents
  4. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedback
  5. 2026Multi-Agent LLMs for Adaptive Acquisition in Bayesian Optimization
  6. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  7. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities
  8. 2026What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search