mlsanityData to 7 Oct 2026

DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints

arXiv 2601.18137 ↗First posted 26 Jan 2026cs.AI cs.CLcited by 43

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

While agent evaluation has shifted toward long-horizon tasks, most benchmarks still emphasize local, step-level reasoning rather than the global constrained optimization (e.g., time and financial budgets) that demands genuine planning ability. Meanwhile, existing LLM planning benchmarks underrepresent the active information gathering and fine-grained local constraints typical of real-world settings. To address this, we introduce DeepPlanning, a challenging benchmark for practical long-horizon agent planning. It features multi-day travel planning and multi-product shopping tasks that require proactive information acquisition, local constrained reasoning, and global constrained optimization. Evaluations on DeepPlanning show that even frontier agentic LLMs struggle with these problems, highlighting the importance of reliable explicit reasoning patterns and parallel tool use for achieving better effectiveness-efficiency trade-offs. Error analysis further points to promising directions for improving agentic LLMs over long planning horizons. We open-source the code and data to support future research.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 17 of its 26 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

17 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models8 Aug
  2. 2025UserBench: An Interactive Gym Environment for User-Centric Agents29 Jul
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  4. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation8 Feb
  5. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb

Other references · 12

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  6. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  7. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  8. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  9. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  10. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  11. 2025TripTailor: A Real-World Benchmark for Personalized Travel Planningcited by 22
  12. 2025TCP: a Benchmark for Temporal Constraint-Based Planningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 43.

43 papers.

Sort:

By area:

  1. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  2. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 49
  3. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 23
  4. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  5. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  6. 2026JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolutioncited by 10
  7. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 6
  8. 2026PM-Bench: Evaluating Prospective Memory in LLM Agentscited by 6
  9. 2026Harnessing AtomisticSkills for Agentic Atomistic Researchcited by 5
  10. 2026GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planningcited by 5
  11. 2026PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory Refinementcited by 4
  12. 2026EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intentcited by 4
31 more
  1. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  2. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
  3. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  4. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  5. 2026Benchmarking Open-Ended Multi-Agent Coordination in Language Agentscited by 2
  6. 2026What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agentscited by 2
  7. 2026SkillAdam: Stable and Efficient Skill Evolution for Agentscited by 2
  8. 2026GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflowscited by 1
  9. 2026PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Modelscited by 1
  10. 2026Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agentscited by 1
  11. 2026Trip+: Benchmarking Agents in Personalized Interactive Travel Planningcited by 1
  12. 2026CausalDS: Benchmarking Causal Reasoning in Data-Science Agentscited by 1
  13. 2026OmniaBench: Benchmarking General AI Agents Across Diverse Scenarioscited by 1
  14. 2026OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agentscited by 1
  15. 2026AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worldscited by 1
  16. 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
  17. 2026Polished but Unresolved: Identifying Late-Stage Pressure States in Long-Horizon Tool-Use Agentscited by 1
  18. 2026Self-Evolving Multi-Agent Framework for Efficient Decision Making in Real-Time Strategy Scenarioscited by 0
  19. 2026AgentOmnia: Scaling Agentic Models for Full-Scenario Applicationscited by 0
  20. 2026SBCO: Self-Supervised, Verifier-Grounded Harness Optimization For Planning Agentscited by 0
  21. 2026LifePlanner: Evaluating LLM Agents for Geo-spatial Planning with Social Media Datacited by 0
  22. 2026CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic Planningcited by 0
  23. 2026BabelArena: A Large-Scale Multilingual Benchmark for LLM Agentscited by 0
  24. 2026World State Generatorcited by 0
  25. 2026LLM Parkinsonism: Executive-Control Failure, Token-Inefficient Persistence, and an Uncertainty-Aware Global Executive Control Architecture for Autonomous Language-Model Agentscited by 0
  26. 2026Adaptive Consistency Graph for Long-Horizon Agentscited by 0
  27. 2026CompoWorld: Compositional Environment Scaling for General Agentscited by 0
  28. 2026HyperMCTS: Hypergraph-Augmented MCTS for Long-Horizon LLM Agentscited by 0
  29. 2026FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Managementcited by 0
  30. 2026PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agentscited by 0
  31. 2026Harness Evolution Hits a Ceiling: When Weight Training Should Begincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agents
  2. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities
  3. 2026TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios
  4. 2026HiMAP-Travel: Hierarchical Multi-Agent Planning for Long-Horizon Constrained Travel
  5. 2025Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents
  6. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example
  7. 2025ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning
  8. 2025Decompose, Plan in Parallel, and Merge: A Novel Paradigm for Large Language Models based Planning with Multiple Constraints