mlsanityData to 7 Oct 2026

Task and Motion Planning with Large Language Models for Object Rearrangement

arXiv 2303.06247 ↗First posted 10 Mar 2023cs.ROcited by 140

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Multi-object rearrangement is a crucial skill for service robots, and commonsense reasoning is frequently needed in this process. However, achieving commonsense arrangements requires knowledge about objects, which is hard to transfer to robots. Large language models (LLMs) are one potential source of this knowledge, but they do not naively capture information about plausible physical arrangements of the world. We propose LLM-GROP, which uses prompting to extract commonsense knowledge about semantically valid object configurations from an LLM and instantiates them with a task and motion planner in order to generalize to varying scene geometry. LLM-GROP allows us to go from natural-language commands to human-aligned object rearrangement in varied environments. Based on human evaluations, our approach achieves the highest rating while outperforming competitive baselines in terms of success rate while maintaining comparable cumulative action costs. Finally, we demonstrate a practical implementation of LLM-GROP on a mobile manipulator in real-world scenarios. Supplementary materials are available at: https://sites.google.com/view/llm-grop

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. First posted on arXiv 10 Mar 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 37; 2024: 56; 2025: 30; 2026: 17.

140 papers.

Sort:

By area:

  1. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  5. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  6. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  7. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  8. 2023Large Language Models as General Pattern Machinescited by 196
  9. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  10. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  11. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  12. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 125
38 more
  1. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  2. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  3. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  4. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  5. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  6. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  7. 2024Vision Language Models are In-Context Value Learnerscited by 84
  8. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  9. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  10. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  11. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  12. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  13. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  14. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  15. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  16. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoningcited by 49
  17. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  18. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Modelscited by 46
  19. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 43
  20. 2024COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Modelscited by 43
  21. 2023Creative Robot Tool Use with Large Language Modelscited by 42
  22. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  23. 2023CoPAL: Corrective Planning of Robot Actions with Large Language Modelscited by 39
  24. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  25. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  26. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  27. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 29
  28. 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
  29. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  30. 2023SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphscited by 24
  31. 2024RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulationcited by 24
  32. 2023Grounding Classical Task Planners via Vision-Language Modelscited by 23
  33. 2024Tools Fail: Detecting Silent Errors in Faulty Toolscited by 23
  34. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
  35. 2023Conformal Temporal Logic Planning using Large Language Modelscited by 21
  36. 2024LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actionscited by 21
  37. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  38. 2024Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfactioncited by 19

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learn from the Past: Language-conditioned Object Rearrangement with Large Language Models
  2. 2025MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoning
  3. 2025A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
  4. 2022Open-vocabulary Queryable Scene Representations for Real World Planning
  5. 2026Language Movement Primitives: Grounding Language Models in Robot Motion
  6. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  7. 2026Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning ↗
  8. 2025Physically Ground Commonsense Knowledge for Articulated Object Manipulation with Analytic Concepts