mlsanityData to 7 Oct 2026

Grounding Classical Task Planners via Vision-Language Models

arXiv 2304.08587 ↗First posted 17 Apr 2023cs.ROcited by 23

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Classical planning systems have shown great advances in utilizing rule-based human knowledge to compute accurate plans for service robots, but they face challenges due to the strong assumptions of perfect perception and action executions. To tackle these challenges, one solution is to connect the symbolic states and actions generated by classical planners to the robot's sensory observations, thus closing the perception-action loop. This research proposes a visually-grounded planning framework, named TPVQA, which leverages Vision-Language Models (VLMs) to detect action failures and verify action affordances towards enabling successful plan execution. Results from quantitative experiments show that TPVQA surpasses competitive baselines from previous studies in task completion rate.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 39 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

29 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  2. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  3. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models23 Jul

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 411
14 more
  1. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  2. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  3. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  4. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  5. 2023Vision-Language Models as Success Detectorscited by 138
  6. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  7. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  8. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  9. 2018Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systemscited by 40
  10. 2022Robot Task Planning and Situation Handling in Open Worldscited by 10
  11. 2020Task-Motion Planning for Safe and Efficient Urban Drivingcited by 7
  12. 2022Visually Grounded Task and Motion Planning for Mobile Manipulationcited by 5
  13. 2022Learning to Ground Objects for Robot Task and Motion Planningcited by 2
  14. 2022GLAD: Grounded Layered Autonomous Driving for Complex Service Taskscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 8; 2026: 4.

23 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  4. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  5. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Modelscited by 46
  6. 2024ReplanVLM: Replanning Robotic Tasks with Visual Language Modelscited by 28
  7. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  8. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  9. 2025A Survey on Large Language Models for Automated Planningcited by 17
  10. 2024Object-Centric Instruction Augmentation for Robotic Manipulationcited by 16
  11. 2025VerifyLLM: LLM-Based Pre-Execution Task Plan Verification for Robotscited by 14
  12. 2024Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challengecited by 9
11 more
  1. 2024GameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Gamescited by 7
  2. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Modelscited by 5
  3. 2025Simulation to Rules: A Dual-VLM Framework for Formal Visual Planningcited by 4
  4. 2026SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Modelscited by 3
  5. 2025RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recoverycited by 2
  6. 2025Using VLM Reasoning to Constrain Task and Motion Planningcited by 2
  7. 2026Tactile Modality Fusion for Vision-Language-Action Modelscited by 2
  8. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  9. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarkscited by 0
  10. 2026SCOPE: Evolving Symbolic World for Planning in Open-Ended Environmentscited by 0
  11. 2026Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bridging Learned Visual Perception and Symbolic Belief-Space Planning
  2. 2026ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback ↗
  3. 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models
  4. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  5. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning
  6. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  7. 2025Vision-Language-Policy Model for Dynamic Robot Task Planning
  8. 2026H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model