mlsanityData to 7 Oct 2026

Plancraft: an evaluation dataset for planning with LLM agents

arXiv 2412.21033 ↗First posted 30 Dec 2024cs.CL cs.AIcited by 16

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We present Plancraft, a multi-modal evaluation dataset for LLM agents. Plancraft has both a text-only and multi-modal interface, based on the Minecraft crafting GUI. We include the Minecraft Wiki to evaluate tool use and Retrieval Augmented Generation (RAG), as well as a handcrafted planner and Oracle Retriever, to ablate the different components of a modern agent architecture. To evaluate decision-making, Plancraft also includes a subset of examples that are intentionally unsolvable, providing a realistic challenge that requires the agent not only to complete tasks but also to decide whether they are solvable at all. We benchmark both open-source and closed-source LLMs and compare their performance and efficiency to a handcrafted planner. Overall, we find that LLMs and VLMs struggle with the planning problems that Plancraft introduces, and offer suggestions on how to improve their capabilities.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 34 of its 44 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

34 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  3. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  4. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents3 Feb

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
18 more
  1. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  8. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  9. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  10. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  11. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  12. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  13. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  14. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  15. 2022TALM: Tool Augmented Language Modelscited by 197
  16. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  17. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  18. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 130

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025Towards a Science of Scaling Agent Systemscited by 125
  2. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  3. 2025What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignmentcited by 7
  4. 2025HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worldscited by 6
  5. 2026Neuro-Symbolic Synergy for World Modelingcited by 6
  6. 2026Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agentscited by 6
  7. 2026Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agentscited by 4
  8. 2026TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systemscited by 4
  9. 2026Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrationscited by 4
  10. 2026Acting Flatterers via LLMs Sycophancy: Combating Clickbait with LLMs Opposing-Stance Reasoningcited by 2
  11. 2025How2: How to learn from procedural How-to questionscited by 1
  12. 2026LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?cited by 1
4 more
  1. 2026Agent Meltdowns: The Road to Hell Is Paved with Helpful Agentscited by 1
  2. 2026ProPlay: Procedural World Models for Self-Evolving LLM Agentscited by 1
  3. 2026Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attributioncited by 0
  4. 2026Object-Centric Environment Modeling for Agentic Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Survey on Evaluation of LLM-based Agents
  2. 2025Evaluation and Benchmarking of LLM Agents: A Survey
  3. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilities
  4. 2026PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents
  5. 2026Toward Architecture-Aware Evaluation Metrics for LLM Agents
  6. 2024Understanding the planning of LLM agents: A survey
  7. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
  8. 2024A LLM Benchmark based on the Minecraft Builder Dialog Agent Task