mlsanityData to 7 Oct 2026

DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning

arXiv 2406.17659 ↗First posted 25 Jun 2024cs.AI cs.ROcited by 14

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have been applied to robot task planning problems, where the robot receives a task in natural language and generates plans based on visual inputs. While current VLMs have demonstrated strong vision-language understanding capabilities, their performance is still far from being satisfactory in planning tasks. At the same time, although classical task planners, such as PDDL-based, are strong in planning for long-horizon tasks, they do not work well in open worlds where unforeseen situations are common. In this paper, we propose a novel task planning and execution framework, called DKPROMPT, which automates VLM prompting using domain knowledge in PDDL for classical planning in open worlds. Results from quantitative experiments show that DKPROMPT outperforms classical planning, pure VLM-based and a few other competitive baselines in task completion rate.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 75 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

53 of 75 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models7 Apr
  2. 2024CognitiveDog: Large Multimodal Model Based System to Translate Vision and Language into Action of Quadruped Robot17 Jan
  3. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration20 Nov
  4. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  5. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  6. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models23 Jul

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
35 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  5. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  8. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  9. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  10. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  11. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  12. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  13. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  14. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  15. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  16. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  17. 2023Vision-Language Models as Success Detectorscited by 138
  18. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  19. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  20. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  21. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  22. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  23. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  24. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  25. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  26. 2022Plansformer: Generating Symbolic Plans using Transformerscited by 41
  27. 2018Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systemscited by 40
  28. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  29. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  30. 2021Grounding Predicates through Actionscited by 25
  31. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  32. 2023Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Modelscited by 10
  33. 2020Task-Motion Planning for Safe and Efficient Urban Drivingcited by 7
  34. 2022Visually Grounded Task and Motion Planning for Mobile Manipulationcited by 5
  35. 2022Learning to Ground Objects for Robot Task and Motion Planningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 4.

14 papers.

Sort:

By area:

  1. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  2. 2024AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robotscited by 7
  3. 2024VeriGraph: Scene Graphs for Execution Verifiable Robot Planningcited by 7
  4. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulationcited by 6
  5. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  6. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Modelscited by 5
  7. 2025Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulationcited by 4
  8. 2025Robot Operation of Home Appliances by Reading User Manualscited by 4
  9. 2025GraSP-VLA: Graph-based Symbolic Action Representation for Long-Horizon Planning with VLA Policiescited by 3
  10. 2024BeSimulator: A Large Language Model Powered Text-based Behavior Simulatorcited by 1
  11. 2025Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided and Self-Consistent MLLMs for Task Planning in Instruction-Following Manipulationcited by 0
  12. 2026Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangementcited by 0
2 more
  1. 2026Bridging Learned Visual Perception and Symbolic Belief-Space Planningcited by 0
  2. 2026Test-Time Spatial Reasoning for Robot Manipulation Using Generative Real-to-Simcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  2. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  3. 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models
  4. 2025Vision-Language-Policy Model for Dynamic Robot Task Planning
  5. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  6. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  7. 2026ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback ↗
  8. 2025Planning with Vision-Language Models and a Use Case in Robot-Assisted Teaching