mlsanityData to 7 Oct 2026

Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

arXiv 2408.07199 ↗First posted 13 Aug 2024cs.AI cs.LGcited by 201

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities in natural language tasks requiring complex reasoning, yet their application in agentic, multi-step reasoning within interactive environments remains a difficult challenge. Traditional supervised pre-training on static datasets falls short in enabling autonomous agent capabilities needed to perform complex decision-making in dynamic settings like web navigation. Previous attempts to bridge this ga-through supervised fine-tuning on curated expert demonstrations-often suffer from compounding errors and limited exploration data, resulting in sub-optimal policy outcomes. To overcome these challenges, we propose a framework that combines guided Monte Carlo Tree Search (MCTS) search with a self-critique mechanism and iterative fine-tuning on agent interactions using an off-policy variant of the Direct Preference Optimization (DPO) algorithm. Our method allows LLM agents to learn effectively from both successful and unsuccessful trajectories, thereby improving their generalization in complex, multi-step reasoning tasks. We validate our approach in the WebShop environment-a simulated e-commerce platform where it consistently outperforms behavior cloning and reinforced fine-tuning baseline, and beats average human performance when equipped with the capability to do online search. In real-world booking scenarios, our methodology boosts Llama-3 70B model's zero-shot performance from 18.6% to 81.7% success rate (a 340% relative increase) after a single day of data collection and further to 95.4% with online search. We believe this represents a substantial leap forward in the capabilities of autonomous agents, paving the way for more sophisticated and reliable decision-making in real-world settings.

Key facts. Cited by 201 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 41 of its 67 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

41 of 67 references.

References · 41

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
29 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  7. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  8. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  9. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  13. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  14. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  15. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  16. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  17. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  18. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  19. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  20. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  21. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  22. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  23. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  24. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  25. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  26. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  27. 2019Meta reinforcement learning as task inferencecited by 75
  28. 2020Human-Level Performance in No-Press Diplomacy via Equilibrium Searchcited by 41
  29. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 111; 2026: 66.

201 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  3. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  5. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  6. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  7. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  8. 2025Agentic Reinforced Policy Optimizationcited by 166
  9. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  10. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  11. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  12. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
38 more
  1. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  2. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  4. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  5. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
  6. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  7. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  8. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  9. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  10. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  11. 2024GUI Agents: A Surveycited by 101
  12. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  13. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  14. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  15. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  16. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  17. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  18. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  19. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  20. 2025PaSa: An LLM Agent for Comprehensive Academic Paper Searchcited by 61
  21. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  22. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 56
  23. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  24. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  25. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  26. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 46
  27. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 45
  28. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  29. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  30. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  31. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  32. 2025A Survey of Direct Preference Optimizationcited by 35
  33. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  34. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  35. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  36. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  37. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  38. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 31

50 most cited of 201 All 201 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
  2. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models
  3. 2024Tree Search for Language Model Agents
  4. 2023A Survey on Large Language Model based Autonomous Agents
  5. 2025DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
  6. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents
  7. 2024Fleet of Agents: Coordinated Problem Solving with Large Language Models
  8. 2025ATLaS: Agent Tuning via Learning Critical Steps