mlsanityData to 7 Oct 2026

Embodied Task Planning with Large Language Models

arXiv 2307.01848 ↗First posted 4 Jul 2023cs.CV cs.AI cs.ROcited by 84

Not on the map. Why?

Cited by 25 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Equipping embodied agents with commonsense is important for robots to successfully complete complex human instructions in general environments. Recent large language models (LLM) can embed rich semantic knowledge for agents in plan generation of complex tasks, while they lack the information about the realistic world and usually yield infeasible action sequences. In this paper, we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models. Specifically, we first construct a multimodal dataset containing triplets of indoor scenes, instructions and action plans, where we provide the designed prompts and the list of existing objects in the scene for GPT-3.5 to generate a large number of instructions and corresponding planned actions. The generated data is leveraged for grounded plan tuning of pre-trained LLMs. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin, which indicates the practicality of embodied task planning in general and complex environments.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

44 of 52 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
32 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2023Instruction Tuning with GPT-4cited by 676
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  13. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  14. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  15. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  16. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  17. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  18. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  19. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  20. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  21. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  22. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  23. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  24. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  25. 2022NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Taskscited by 60
  26. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathologycited by 57
  27. 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
  28. 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
  29. 2022Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancementcited by 17
  30. 2022GE-Grasp: Efficient Target-Oriented Grasping in Dense Cluttercited by 8
  31. 2022Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Explorationcited by 5
  32. 20233D Small Object Detection with Dynamic Spatial Pruningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 25; 2025: 33; 2026: 13.

84 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  5. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  6. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  7. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  8. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  9. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  10. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  11. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  12. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
38 more
  1. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  2. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  3. 2024Tool-Planner: Task Planning with Clusters across Multiple Toolscited by 32
  4. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  5. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 26
  6. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  7. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  8. 2023SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Datacited by 24
  9. 2024RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulationcited by 24
  10. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24
  11. 2025Embodied AI: From LLMs to World Modelscited by 23
  12. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  13. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  14. 2023Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMscited by 19
  15. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  16. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 19
  17. 2023LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systemscited by 17
  18. 2024AutoGPT+P: Affordance-based Task Planning with Large Language Modelscited by 17
  19. 2025NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domainscited by 15
  20. 2025Learning Compositional Behaviors from Demonstration and Languagecited by 15
  21. 2024Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Frameworkcited by 13
  22. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  23. 2025Reinforced Reasoning for Embodied Planningcited by 12
  24. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  25. 2025Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planningcited by 10
  26. 2023MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memorycited by 9
  27. 2024Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Modelscited by 9
  28. 2024Embodied CoT Distillation From LLM To Off-the-shelf Agentscited by 9
  29. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Plannercited by 9
  30. 2024Embodied Instruction Following in Unknown Environmentscited by 8
  31. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  32. 2024Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Followingcited by 7
  33. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?cited by 7
  34. 2025From reactive to cognitive: brain-inspired spatial intelligence for embodied agentscited by 7
  35. 2025Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Followingcited by 7
  36. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  37. 2024Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Modelscited by 6
  38. 2025CLEA: Closed-Loop Embodied Agent for Enhancing Task Execution in Dynamic Environmentscited by 6

50 most cited of 84 All 84 (JSON)