mlsanityData to 7 Oct 2026

Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

arXiv 2302.01560 ↗First posted 3 Feb 2023cs.AIcited by 396

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose "underlineDescribe, underlineExplain, underlinePlan and underlineSelect" (DEPS), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated textitplan by integrating textitdescription of the plan execution process and providing self-textitexplanation of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal textitselector, which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the textttObtainDiamond grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner.

Key facts. Cited by 396 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 36 of its 66 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

36 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022A Generalist Agent12 May

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
23 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2016The Option-Critic Architecturecited by 433
  12. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  13. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  14. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  15. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  16. 2023Collaborating with language models for embodied reasoningcited by 61
  17. 2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 40
  18. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  19. 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 35
  20. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  21. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  22. 2019The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 25
  23. 2021SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competitioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 104; 2024: 140; 2025: 96; 2026: 56.

396 papers.

Sort:

By area:

  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  6. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  8. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  9. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  10. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  11. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  12. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
38 more
  1. 2023Augmented Language Models: a Surveycited by 398
  2. 2023Language Models can Solve Computer Taskscited by 371
  3. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  4. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  5. 2023Tool Learning with Foundation Modelscited by 302
  6. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  7. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  10. 2023Challenges and Applications of Large Language Modelscited by 251
  11. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  12. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  13. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  14. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  15. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  16. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  17. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  18. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  19. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  20. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  21. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  22. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  23. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  24. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  25. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  26. 2023Video Language Planningcited by 160
  27. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  28. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  29. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  30. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  31. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  32. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  33. 2023Large Language Models for Robotics: A Surveycited by 132
  34. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
  35. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 129
  36. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  37. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  38. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123

50 most cited of 396 All 396 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints
  2. 2024TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models
  3. 2024Multi-Agent Planning Using Visual Language Models
  4. 2026From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents
  5. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
  6. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks
  7. 2025Constrained Natural Language Action Planning for Resilient Embodied Systems
  8. 2023ADaPT: As-Needed Decomposition and Planning with Language Models