mlsanityData to 14 Oct 2026

Text2Motion: From Natural Language Instructions to Feasible Plans

arXiv 2303.12153 ↗First posted 21 Mar 2023cs.RO cs.AI cs.LGcited by 224

On the map: World models → Embodied and robot world models

Abstract (arXiv)

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills.

Key facts. Cited by 224 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 55 of its 101 references that are in the map. First posted on arXiv 21 Mar 2023.

Builds on

55 of 101 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2017On Calibration of Modern Neural Networkscited by 4,960
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  7. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  10. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  13. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  14. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  15. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  16. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  17. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  18. 2020Integrated Task and Motion Planningcited by 290
  19. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  20. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 178
  21. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  22. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 150
  23. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  24. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  25. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  26. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  27. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  28. 2021Learning Symbolic Operators for Task and Motion Planningcited by 58
  29. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 58
  30. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  31. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 48
  32. 2020Learning compositional models of robot skills for task and motion planningcited by 48
  33. 2021Learning Neuro-Symbolic Relational Transition Models for Bilevel Planningcited by 43
  34. 2020Deep Affordance Foresight: Planning Through What Can Be Done in the Futurecited by 41
  35. 2021Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordancescited by 40
  36. 2020Planning with Learned Object Importance in Large Problem Instances using Graph Neural Networkscited by 38
  37. 2018Active model learning and diverse action sampling for task and motion planningcited by 28
  38. 2020CAMPs: Learning Context-Specific Abstractions for Efficient Planning in Factored MDPscited by 18

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 60; 2024: 83; 2025: 55; 2026: 26.

224 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  2. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  3. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  7. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  9. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  10. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  11. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 207
  12. 2023Large Language Models as General Pattern Machinescited by 197
38 more
  1. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  2. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  3. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  4. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  5. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  6. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  7. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  8. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 138
  9. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  10. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  11. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  12. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 125
  13. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  14. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 113
  15. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 111
  16. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  17. 2023Compositional Foundation Models for Hierarchical Planningcited by 95
  18. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  19. 2024Vision Language Models are In-Context Value Learnerscited by 90
  20. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  21. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  22. 2024Coercing LLMs to do and reveal (almost) anythingcited by 89
  23. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 83
  24. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 83
  25. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  26. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 77
  27. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  28. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  29. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 70
  30. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 68
  31. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  32. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  33. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  34. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
  35. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  36. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Modelscited by 47
  37. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 45
  38. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45

50 most cited of 224 All 224 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation
  2. 2025Reinforced Embodied Planning with Verifiable Reward for Real-World Robotic Manipulation
  3. 2026Thinking in Text and Images: Interleaved Vision--Language Reasoning Traces for Long-Horizon Robot Manipulation
  4. 2025A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
  5. 2025Lang2Manip: A Tool for LLM-Based Symbolic-to-Geometric Planning for Manipulation
  6. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  7. 2024Open-World Task and Motion Planning via Vision-Language Model Generated Constraints
  8. 2024Points2Plans: From Point Clouds to Long-Horizon Plans with Composable Relational Dynamics