mlsanityData to 7 Oct 2026

Interactive Task Planning with Language Models

arXiv 2310.10645 ↗First posted 16 Oct 2023cs.RO cs.AI cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals and distinct tasks, even during execution. However, most traditional methods require predefined module design, making it hard to generalize to different goals. Recent large language model based approaches can allow for more open-ended planning but often require heavy prompt engineering or domain specific pretrained models. To tackle this, we propose a simple framework that achieves interactive task planning with language models by incorporating both high-level planning and low-level skill execution through function calling, leveraging pretrained vision models to ground the scene in language. We verify the robustness of our system on the real world task of making milk tea drinks. Our system is able to generate novel high-level instructions for unseen objectives and successfully accomplishes user tasks. Furthermore, when the user sends a new request, our system is able to replan accordingly with precision based on the new request, task guidelines and previously executed steps. Our approach is easy to adapt to different tasks by simply substituting the task guidelines, without the need for additional complex prompt engineering. Please check more details on our https://wuphilipp.github.io/itp_site and https://youtu.be/TrKLuyv26_g.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

22 of 42 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  12. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
10 more
  1. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  2. 2020Integrated Task and Motion Planningcited by 281
  3. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  4. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  5. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  6. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  7. 2023Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Promptingcited by 45
  8. 2018Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systemscited by 40
  9. 2016FFRob: Leveraging Symbolic Planning for Efficient Task and Motion Planningcited by 34
  10. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 12; 2026: 7.

36 papers.

Sort:

By area:

  1. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  2. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  3. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  4. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  5. 2024From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Controlcited by 34
  6. 2024ReplanVLM: Replanning Robotic Tasks with Visual Language Modelscited by 28
  7. 2024An Interactive Agent Foundation Modelcited by 25
  8. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  9. 2024A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddingscited by 23
  10. 2024Tools Fail: Detecting Silent Errors in Faulty Toolscited by 23
  11. 2025Drift No More? Context Equilibria in Multi-Turn LLM Interactionscited by 21
  12. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
24 more
  1. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  2. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  3. 2024Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Gamescited by 8
  4. 2025REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulationcited by 8
  5. 2024VeriGraph: Scene Graphs for Execution Verifiable Robot Planningcited by 7
  6. 2025Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Followingcited by 7
  7. 2024Creative Problem Solving in Large Language and Vision Models -- What Would it Take?cited by 6
  8. 2024Unifying Large Language Model and Deep Reinforcement Learning for Human-in-Loop Interactive Socially-aware Navigationcited by 4
  9. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  10. 2025Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scenecited by 4
  11. 2025Human-like Navigation in a World Built for Humanscited by 4
  12. 2026Hippocampus: An Efficient and Scalable Memory Module for Agentic AIcited by 3
  13. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  14. 2024Position Paper: Agent AI Towards a Holistic Intelligencecited by 2
  15. 2024Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programmingcited by 2
  16. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  17. 2026How to Steer Your Multi-Agent System: Human-LLM Collaborative Planningcited by 2
  18. 2024LLM-guided Task and Motion Planning using Knowledge-based Reasoningcited by 1
  19. 2025From Code to Action: Hierarchical Learning of Diffusion-VLM Policiescited by 1
  20. 2025Information Seeking for Robust Decision Making under Partial Observabilitycited by 1
  21. 2025Real-Time World Crafting: Generating Structured Game Behaviors from Natural Language with Large Language Modelscited by 1
  22. 2026Make Your VLA More Robust Without More Data By Interleaving Motion Planningcited by 1
  23. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0
  24. 2026Safe Task Planning with Long-Term Graph Memory for Embodied Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Models
  2. 2025Vision-Language-Policy Model for Dynamic Robot Task Planning
  3. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  4. 2026Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning ↗
  5. 2025Leveraging Pre-trained Large Language Models with Refined Prompting for Online Task and Motion Planning
  6. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  7. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  8. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning