mlsanityData to 7 Oct 2026

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

arXiv 2402.01622 ↗First posted 2 Feb 2024cs.CLcited by 385

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.

Key facts. Cited by 385 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 45 of its 69 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

45 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  2. 2023Cognitive Architectures for Language Agents5 Sep
  3. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  4. 2022Language Models as Agent Models3 Dec

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
29 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2024Mixtral of Expertscited by 1,785
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  13. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  14. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  15. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  16. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  17. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  18. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  19. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  20. 2016Solving General Arithmetic Word Problemscited by 353
  21. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  22. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  23. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  24. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  25. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  26. 2016The Predictron: End-To-End Learning and Planningcited by 92
  27. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  28. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  29. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 78; 2025: 170; 2026: 137.

385 papers.

Sort:

By area:

  1. 2024AFlow: Automating Agentic Workflow Generationcited by 395
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  4. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  7. 2024AI Agents That Mattercited by 199
  8. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  9. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  10. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 141
  11. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 125
  12. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 114
38 more
  1. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 109
  2. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  3. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  4. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  5. 2024Large Multimodal Agents: A Surveycited by 99
  6. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  7. 2024AIOS: LLM Agent Operating Systemcited by 94
  8. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 93
  9. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 91
  10. 2025Memp: Exploring Agent Procedural Memorycited by 88
  11. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  12. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  13. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  14. 2025Agent Learning via Early Experiencecited by 79
  15. 2026Privileged Information Distillation for Language Modelscited by 79
  16. 2024AutoFlow: Automated Workflow Generation for Large Language Model Agentscited by 76
  17. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 73
  18. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 72
  19. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
  20. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 60
  21. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 59
  22. 2025Evolving Deeper LLM Thinkingcited by 58
  23. 2025Self-Challenging Language Model Agentscited by 56
  24. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  25. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 51
  26. 2024Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Toolscited by 47
  27. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  28. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  29. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  30. 2024Evaluating Cultural and Social Awareness of LLM Web Agentscited by 43
  31. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 43
  32. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
  33. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  34. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  35. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  36. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  37. 2024TravelAgent: An AI Assistant for Personalized Travel Planningcited by 38
  38. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38

50 most cited of 385 All 385 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Itinerary Planning-A Real-World Benchmark for Multi-Turn and Tool-Using Travel Tasks
  2. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
  3. 2025ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning
  4. 2025LLMs Can Plan Only If We Tell Them
  5. 2024Query-Efficient Planning with Language Models
  6. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks
  7. 2026GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI
  8. 2023Translating Natural Language to Planning Goals with Large-Language Models