mlsanityData to 7 Oct 2026

LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning

arXiv 2403.11552 ↗First posted 18 Mar 2024cs.RO cs.AIcited by 63

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Conventional Task and Motion Planning (TAMP) approaches rely on manually crafted interfaces connecting symbolic task planning with continuous motion generation. These domain-specific and labor-intensive modules are limited in addressing emerging tasks in real-world settings. Here, we present LLM^3, a novel Large Language Model (LLM)-based TAMP framework featuring a domain-independent interface. Specifically, we leverage the powerful reasoning and planning capabilities of pre-trained LLMs to propose symbolic action sequences and select continuous action parameters for motion planning. Crucially, LLM^3 incorporates motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failure. Consequently, LLM^3 interfaces between task planning and motion planning, alleviating the intricate design process of handling domain-specific messages between them. Through a series of simulations in a box-packing domain, we quantitatively demonstrate the effectiveness of LLM^3 in solving TAMP problems and the efficiency in selecting action parameters. Ablation studies underscore the significant contribution of motion failure reasoning to the success of LLM^3. Furthermore, we conduct qualitative experiments on a physical manipulator, demonstrating the practical applicability of our approach in real-world settings.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 54 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

39 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Task and Motion Planning with Large Language Models for Object Rearrangement10 Mar
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  12. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
25 more
  1. 2020Integrated Task and Motion Planningcited by 281
  2. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  3. 2023Large Language Models as General Pattern Machinescited by 196
  4. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  5. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  6. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  7. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  8. 2023MindAgent: Emergent Gaming Interactioncited by 87
  9. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  10. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  11. 2022Predicate Invention for Bilevel Planningcited by 49
  12. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  13. 2019VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learningcited by 29
  14. 2018Active model learning and diverse action sampling for task and motion planningcited by 28
  15. 2023DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observabilitycited by 21
  16. 2022Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planningcited by 19
  17. 2022Sequential Manipulation Planning on Scene Graphcited by 16
  18. 2020Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Taskscited by 15
  19. 2022Transformers are Adaptable Task Plannerscited by 15
  20. 2021Active Learning of Abstract Plan Feasibilitycited by 11
  21. 2021Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspectivecited by 11
  22. 2021Consolidating Kinematic Models to Promote Coordinated Mobile Manipulationscited by 8
  23. 2023LEMMA: Learning Language-Conditioned Multi-Robot Manipulationcited by 7
  24. 2023Sequential Manipulation Planning for Over-actuated Unmanned Aerial Manipulatorscited by 2
  25. 2023Learning Concept-Based Causal Transition and Symbolic Reasoning for Visual Planningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 26; 2026: 22.

63 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  3. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 72
  4. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Modelscited by 46
  5. 2024LLM-A*: Large Language Model Enhanced Incremental Heuristic Search on Path Planningcited by 44
  6. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  7. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  8. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 31
  9. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 28
  10. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 23
  11. 2025Embodied AI: From LLMs to World Modelscited by 23
  12. 2024Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfactioncited by 19
38 more
  1. 2024Large Language Models as Software Components: A Taxonomy for LLM-Integrated Applicationscited by 17
  2. 2024Explore the Reasoning Capability of LLMs in the Chess Testbedcited by 17
  3. 2025A Survey on Large Language Models for Automated Planningcited by 17
  4. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14
  5. 2024Open-World Task and Motion Planning via Vision-Language Model Generated Constraintscited by 9
  6. 2025REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulationcited by 8
  7. 2025R^3-VQA: "Read the Room" by Video Social Reasoningcited by 6
  8. 2026ALRM: Agentic LLM for Robotic Manipulationcited by 6
  9. 2026H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Modelcited by 6
  10. 2024M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenescited by 5
  11. 2025Robot Operation of Home Appliances by Reading User Manualscited by 4
  12. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  13. 2026From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Modelscited by 4
  14. 2025Vision Language Models Cannot Plan, but Can They Formalize?cited by 2
  15. 2025Unifying Deep Predicate Invention with Pre-trained Foundation Modelscited by 2
  16. 2026Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planningcited by 2
  17. 2026SAIL: Test-Time Scaling for In-Context Imitation Learning with VLMcited by 2
  18. 2024LLM-guided Task and Motion Planning using Knowledge-based Reasoningcited by 1
  19. 2024TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applicationscited by 1
  20. 2025AINav: Large Language Model-Based Adaptive Interactive Navigationcited by 1
  21. 2025Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actionscited by 1
  22. 2025RoboPARA: Dual-Arm Robot Planning with Parallel Allocation and Recomposition Across Taskscited by 1
  23. 2025PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulationcited by 1
  24. 2025An LLM-powered Natural-to-Robotic Language Translation Framework with Correctness Guaranteescited by 1
  25. 2025SDA-PLANNER: State-Dependency Aware Adaptive Planner for Embodied Task Planningcited by 1
  26. 2025Information Seeking for Robust Decision Making under Partial Observabilitycited by 1
  27. 2026Analogical Trajectory Transfercited by 1
  28. 2026IMBench: A Benchmark for Intuitive Robotic Manipulationcited by 1
  29. 2026Coding Agents for Generalized Task and Motion Planning Problemscited by 1
  30. 2025Bidirectional Task-Motion Planning Based on Hierarchical Reinforcement Learning for Strategic Confrontationcited by 0
  31. 2025From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Controlcited by 0
  32. 2025Grounded Vision-Language Interpreter for Long-Horizon Bimanual Task and Motion Planningcited by 0
  33. 2025Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planningcited by 0
  34. 2025CoFineLLM: Conformal Finetuning of LLMs for Language-Instructed Robot Planningcited by 0
  35. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarkscited by 0
  36. 2025Prompt2Craft: Generating Functional Craft Assemblies with LLMscited by 0
  37. 2025LADY: Linear Attention for Autonomous Driving Efficiency without Transformerscited by 0
  38. 2026MMP-A*: Multimodal Perception Enhanced Incremental Heuristic Search on Path Planningcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
  2. 2025One Demo Is All It Takes: Planning Domain Derivation with LLMs from A Single Demonstration
  3. 2025T3 Planner: A Self-Correcting LLM Framework for Robotic Motion Planning with Temporal Logic
  4. 2025Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generation
  5. 2025Plan Verification for LLM-Based Embodied Task Completion Agents
  6. 2026GAVEL: Graph World Models for Verified and Efficient Long-Horizon LLM Task Planning
  7. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  8. 2023Dynamic Planning with a LLM