mlsanityData to 7 Oct 2026

On Grounded Planning for Embodied Tasks with Language Models

arXiv 2209.00465 ↗First posted 29 Aug 2022cs.AI cs.CL cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) have demonstrated their capability in possessing commonsense knowledge of the physical world, a crucial aspect of performing tasks in everyday life. However, it remains unclear **whether LMs have the capacity to generate grounded, executable plans for embodied tasks.** This is a challenging task as LMs lack the ability to perceive the environment through vision and feedback from the physical environment. In this paper, we address this important research question and present the first investigation into the topic. Our novel problem formulation, named **G-PlanET**, inputs a high-level goal and a data table about objects in a specific environment, and then outputs a step-by-step actionable plan for a robotic agent to follow. To facilitate the study, we establish an **evaluation protocol** and design a dedicated metric to assess the quality of the plans. Our experiments demonstrate that the use of tables for encoding the environment and an iterative decoding strategy can significantly enhance the LMs' ability in grounded planning. Our analysis also reveals interesting and non-trivial findings.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 29 Aug 2022.

Builds on

26 of 36 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
14 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  2. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  3. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  4. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  5. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  6. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  7. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  8. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  9. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  10. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  11. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  12. 2022Reasoning Like Program Executorscited by 45
  13. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  14. 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 14; 2025: 5; 2026: 2.

33 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  3. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  4. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  5. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  6. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  7. 2023Penetrative AI: Making LLMs Comprehend the Physical Worldcited by 64
  8. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  9. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agentscited by 51
  10. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  11. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  12. 2023Interactive Task Planning with Language Modelscited by 36
21 more
  1. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  2. 2024Tool-Planner: Task Planning with Clusters across Multiple Toolscited by 32
  3. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  4. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulationcited by 26
  5. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24
  6. 2025Embodied AI: From LLMs to World Modelscited by 23
  7. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  8. 2023Responsible Task Automation: Empowering Large Language Models as Responsible Task Automatorscited by 14
  9. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
  10. 2025Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verificationcited by 9
  11. 2023Multimodal Pretrained Models for Verifiable Sequential Decision-Making: Planning, Grounding, and Perceptioncited by 8
  12. 2024Look Further Ahead: Testing the Limits of GPT-4 in Path Planningcited by 7
  13. 2024Multi-Agent Planning Using Visual Language Modelscited by 6
  14. 2025Cook and Clean Together: Teaching Embodied Agents for Parallel Task Executioncited by 6
  15. 2024Open Grounded Planning: Challenges and Benchmark Constructioncited by 4
  16. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  17. 2023On the Planning, Search, and Memorization Capabilities of Large Language Modelscited by 1
  18. 2024All Language Models Large and Smallcited by 1
  19. 2024Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environmentscited by 0
  20. 2026Beyond the Text: Verifying That Agent-Written Papers Are Backed by Their Artifactscited by 0
  21. 2026Representation Alignment as a Bottleneck in LLM-Based Retrosynthesis Planningcited by 0