mlsanityData to 7 Oct 2026

Tree Search for Language Model Agents

arXiv 2407.01476 ↗First posted 1 Jul 2024cs.AI cs.CL cs.LGcited by 148

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Autonomous agents powered by language models (LMs) have demonstrated promise in their ability to perform decision-making tasks such as web automation. However, a key limitation remains: LMs, primarily optimized for natural language understanding and generation, struggle with multi-step reasoning, planning, and using environmental feedback when attempting to solve realistic computer tasks. Towards addressing this, we propose an inference-time search algorithm for LM agents to explicitly perform exploration and multi-step planning in interactive web environments. Our approach is a form of best-first tree search that operates within the actual environment space, and is complementary with most existing state-of-the-art agents. It is the first tree search algorithm for LM agents that shows effectiveness on realistic web tasks. On the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%. Our experiments highlight the effectiveness of search for web agents, and we demonstrate that performance scales with increased test-time compute. We conduct a thorough analysis of our results to highlight improvements from search, limitations, and promising directions for future work. Our code and models are publicly released at https://jykoh.com/search-agents.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 45 of its 70 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

45 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct

Other references · 44

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
32 more
  1. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2018World Modelscited by 1,444
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  11. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  12. 2023Language Models can Solve Computer Taskscited by 371
  13. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  14. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  15. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  16. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  17. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  18. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  19. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  20. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  21. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  22. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  23. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  24. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  25. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  26. 2023SteP: Stacked LLM Policies for Web Actionscited by 66
  27. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  28. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 54
  29. 2023SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Modelscited by 53
  30. 2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 46
  31. 2020Human-Level Performance in No-Press Diplomacy via Equilibrium Searchcited by 41
  32. 2024LiteSearch: Efficacious Tree Search for LLMcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 83; 2026: 44.

148 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  4. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  5. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  6. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  7. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  10. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  11. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 124
  12. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
38 more
  1. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  2. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
  3. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  4. 2024GUI Agents: A Surveycited by 101
  5. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  6. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  7. 2025Agent Learning via Early Experiencecited by 79
  8. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  9. 2024Beyond Browsing: API-Based Web Agentscited by 77
  10. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  11. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  12. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  13. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  14. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  15. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  16. 2025LifelongAgentBench: Evaluating LLM Agents as Lifelong Learnerscited by 65
  17. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  18. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  19. 2025Scaling Test-time Compute for LLM Agentscited by 59
  20. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  21. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 56
  22. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  23. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  24. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  25. 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43
  26. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 43
  27. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  28. 2024ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Datacited by 39
  29. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  30. 2025MPO: Boosting LLM Agents with Meta Plan Optimizationcited by 36
  31. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  32. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  33. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  34. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  35. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  36. 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Modelscited by 32
  37. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31
  38. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving LLM-based Autonomous Web Agents with Filtering
  2. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
  3. 2026GATS: Graph-Augmented Tree Search with Layered World Models for Efficient Agent Planning
  4. 2024Fleet of Agents: Coordinated Problem Solving with Large Language Models
  5. 2026Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents
  6. 2023A Survey on Large Language Model based Autonomous Agents
  7. 2026AI Planning Framework for LLM-Based Web Agents
  8. 2025ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task Planning