mlsanityData to 7 Oct 2026

ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

arXiv 2209.11302 ↗First posted 22 Sep 2022cs.RO cs.AI cs.CLcited by 598

Not on the map. Why?

Cited by 132 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at progprompt.github.io

Key facts. Cited by 598 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 22 Sep 2022.

Builds on

30 of 40 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  10. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
18 more
  1. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  2. 2017Challenges in Data-to-Document Generationcited by 226
  3. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  4. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  5. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  6. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  7. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  8. 2021Skill Induction and Planning with Latent Languagecited by 99
  9. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  10. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  11. 2017Neural Task Programming: Learning to Generalize Across Hierarchical Taskscited by 75
  12. 2018Learning Plannable Representations with Causal InfoGANcited by 56
  13. 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43
  14. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  15. 2018Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systemscited by 40
  16. 2020Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructionscited by 36
  17. 2019Regression Planning Networkscited by 32
  18. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 137; 2024: 173; 2025: 142; 2026: 137.

598 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  7. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  8. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  10. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  2. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  3. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  4. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  5. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  6. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  7. 2023Cognitive Architectures for Language Agentscited by 422
  8. 2026World Action Models are Zero-shot Policiescited by 411
  9. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  12. 2023Tool Learning with Foundation Modelscited by 302
  13. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  14. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  15. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  16. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  17. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  18. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  19. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  20. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  21. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  22. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  23. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  24. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  25. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  26. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  27. 2023Large Language Models as General Pattern Machinescited by 196
  28. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  29. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  30. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  31. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  32. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  33. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  34. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  35. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  36. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  37. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  38. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161

50 most cited of 598 All 598 (JSON)