mlsanityData to 14 Oct 2026

LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models

arXiv 2212.04088 ↗First posted 8 Dec 2022cs.AI cs.CL cs.CVcited by 462

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

This study focuses on using large language models (LLMs) as a planner for embodied agents that can follow natural language instructions to complete complex tasks in a visually-perceived environment. The high data cost and poor sample efficiency of existing methods hinders the development of versatile agents that are capable of many tasks and can learn new tasks quickly. In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models to do few-shot planning for embodied agents. We further propose a simple but effective way to enhance LLMs with physical grounding to generate and update plans that are grounded in the current environment. Experiments on the ALFRED dataset show that our method can achieve very competitive few-shot performance: Despite using less than 0.5% of paired training data, LLM-Planner achieves competitive performance with recent baselines that are trained using the full training data. Existing methods can barely complete any task successfully under the same few-shot setting. Our work opens the door for developing versatile and sample-efficient embodied agents that can quickly learn many tasks. Website: https://dki-lab.github.io/LLM-Planner

Key facts. Cited by 462 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 52 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

45 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents28 Aug
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
31 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  5. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  9. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  10. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  12. 2022Visual Language Maps for Robot Navigationcited by 365
  13. 2021True Few-Shot Learning with Language Modelscited by 344
  14. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  15. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  16. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  17. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  18. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  19. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  20. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  21. 2021Skill Induction and Planning with Latent Languagecited by 99
  22. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  23. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  24. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  25. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 58
  26. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  27. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  28. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  29. 2021Language Models are Few-Shot Butlerscited by 27
  30. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  31. 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 70; 2024: 154; 2025: 154; 2026: 82.

462 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  4. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  6. 2023A Comprehensive Overview of Large Language Modelscited by 648
  7. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  8. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  9. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  10. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  11. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  12. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
38 more
  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  3. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  4. 2023Large Language Model Alignment: A Surveycited by 243
  5. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  6. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  7. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  8. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  9. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  10. 2023Large Language Models as General Pattern Machinescited by 197
  11. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  12. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 187
  13. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  14. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  15. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 175
  16. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  17. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 168
  18. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 164
  19. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  20. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  21. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  22. 2023Large Language Models for Robotics: A Surveycited by 133
  23. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 126
  24. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  25. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  26. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  27. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  28. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 116
  29. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 113
  30. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  31. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  32. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
  33. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  34. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 100
  35. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  36. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 96
  37. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  38. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88

50 most cited of 462 All 462 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Dynamic Planning with a LLM
  2. 2024Understanding the planning of LLM agents: A survey
  3. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
  4. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models
  5. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models
  6. 2024Aligning Agents like Large Language Models
  7. 2024Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback
  8. 2025Empowering LLMs with Parameterized Skills for Adversarial Long-Horizon Planning