mlsanityData to 14 Oct 2026

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

arXiv 2201.07207 ↗First posted 18 Jan 2022cs.LG cs.AI cs.CLcited by 1,035

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Can world knowledge learned by large language models (LLMs) be used to act in interactive environments? In this paper, we investigate the possibility of grounding high-level tasks, expressed in natural language (e.g. "make breakfast"), to a chosen set of actionable steps (e.g. "open fridge"). While prior work focused on learning from explicit step-by-step examples of how to act, we surprisingly find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into mid-level plans without any further training. However, the plans produced naively by LLMs often cannot map precisely to admissible actions. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions. Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline. The conducted human evaluation reveals a trade-off between executability and correctness but shows a promising sign towards extracting actionable knowledge from language models. Website at https://huangwl18.github.io/language-planner

Key facts. Cited by 1,035 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 55 references that are in the map. First posted on arXiv 18 Jan 2022.

Builds on

42 of 55 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2019Language Models as Knowledge Bases?cited by 1,936
30 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2019How Can We Know What Language Models Know?cited by 959
  8. 2021Measuring Coding Challenge Competence With APPScited by 902
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  13. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  14. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  15. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  16. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  17. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  18. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  19. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  20. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  21. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  22. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  23. 2021Skill Induction and Planning with Latent Languagecited by 99
  24. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  25. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  26. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  27. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 58
  28. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  29. 2020Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructionscited by 36
  30. 2020Probing Contextual Language Models for Common Ground with Visual Representationscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 66; 2023: 261; 2024: 324; 2025: 226; 2026: 158.

1,035 papers.

Sort:

By area:

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
38 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  8. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  9. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  11. 2022A Generalist Agentcited by 808
  12. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  13. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  14. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  15. 2023A Comprehensive Overview of Large Language Modelscited by 648
  16. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  17. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  18. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  19. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  20. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  21. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  22. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  23. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  24. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  25. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  26. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  27. 2023Cognitive Architectures for Language Agentscited by 429
  28. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  29. 2023Augmented Language Models: a Surveycited by 403
  30. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  31. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  32. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  33. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  34. 2023Language Models can Solve Computer Taskscited by 372
  35. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
  36. 2022Visual Language Maps for Robot Navigationcited by 365
  37. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  38. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353

50 most cited of 1,035 All 1,035 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Grounding Multimodal Large Language Models in Actions
  2. 2023Collaborating with language models for embodied reasoning
  3. 2025Think, Act, Learn: A Framework for Autonomous Robotic Agents using Closed-Loop Large Language Models
  4. 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Models
  5. 2026Embodied Task Planning via Graph-Informed Action Generation with Large Language Models
  6. 2025Constrained Natural Language Action Planning for Resilient Embodied Systems
  7. 2022Few-shot Subgoal Planning with Language Models
  8. 2024Sub-goal Distillation: A Method to Improve Small Language Agents