# ReplanVLM: Replanning Robotic Tasks with Visual Language Models

arXiv: https://arxiv.org/abs/2407.21762 · HTML: https://mlsanity.com/paper/2407.21762/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 24 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 20 of 20)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning](https://mlsanity.com/paper/2305.14909/) (2023)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://mlsanity.com/paper/2311.17842/) (2023)
- [Generalized Planning in PDDL Domains with Pretrained Large Language Models](https://mlsanity.com/paper/2305.11014/) (2023)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [L3MVN: Leveraging Large Language Models for Visual Target Navigation](https://mlsanity.com/paper/2304.05501/) (2023)
- [GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration](https://mlsanity.com/paper/2311.12015/) (2023)
- [Chat with the Environment: Interactive Multimodal Perception Using Large Language Models](https://mlsanity.com/paper/2303.08268/) (2023)
- [RePLan: Robotic Replanning with Perception and Language Models](https://mlsanity.com/paper/2401.04157/) (2024)
- [Interactive Task Planning with Language Models](https://mlsanity.com/paper/2310.10645/) (2023)
- [Large Language Models as Zero-Shot Human Models for Human-Robot Interaction](https://mlsanity.com/paper/2303.03548/) (2023)
- [Vision-Language Interpreter for Robot Task Planning](https://mlsanity.com/paper/2311.00967/) (2023)
- [Grounding Classical Task Planners via Vision-Language Models](https://mlsanity.com/paper/2304.08587/) (2023)
- [GameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Games](https://mlsanity.com/paper/2405.13751/) (2024)
- [Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model](https://mlsanity.com/paper/2308.15684/) (2023)

## Built on by (most cited first, 28 of 28)

- [Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2502.16707/) (2025)
- [Robix: A Unified Model for Robot Interaction, Reasoning and Planning](https://mlsanity.com/paper/2509.01106/) (2025)
- [RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-World](https://mlsanity.com/paper/2412.00171/) (2024)
- [Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models](https://mlsanity.com/paper/2512.01946/) (2025)
- [Large Language Models for Combinatorial Optimization of Design Structure Matrix](https://mlsanity.com/paper/2411.12571/) (2024)
- [ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models](https://mlsanity.com/paper/2508.11918/) (2025)
- [ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models](https://mlsanity.com/paper/2505.13180/) (2025)
- [Compose by Focus: Scene Graph-based Atomic Skills](https://mlsanity.com/paper/2509.16053/) (2025)
- [Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review](https://mlsanity.com/paper/2505.20503/) (2025)
- [ManiAgent: An Agentic Framework for General Robotic Manipulation](https://mlsanity.com/paper/2510.11660/) (2025)
- [Vision-Language-Policy Model for Dynamic Robot Task Planning](https://mlsanity.com/paper/2512.19178/) (2025)
- [Cybo-Waiter: A Physical Agentic Framework for Humanoid Whole-Body Locomotion-Manipulation](https://mlsanity.com/paper/2603.10675/) (2026)
- [ContextFlow: Hierarchical Task-State Alignment for Long-Horizon Embodied Agents](https://mlsanity.com/paper/2605.19314/) (2026)
- [Closing the Loop in Humanoid VLA: Persistent 3D Object Tokens for Verifiable Loco-Manipulation](https://mlsanity.com/paper/2607.18016/) (2026)
- [Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions](https://mlsanity.com/paper/2506.03760/) (2025)
- [A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real World](https://mlsanity.com/paper/2507.16713/) (2025)
- [Multimodal Benchmark for Safety Assessment in Industrial Inspection Scenarios](https://mlsanity.com/paper/2601.21173/) (2026)
- [Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction](https://mlsanity.com/paper/2602.14551/) (2026)
- [Selective Perception for Robot: Task-Aware Attention in Multimodal VLA](https://mlsanity.com/paper/2602.15543/) (2026)
- [MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation](https://mlsanity.com/paper/2602.16898/) (2026)
- [The Robot's Inner Critic: Self-Refinement of Social Behaviors through VLM-based Replanning](https://mlsanity.com/paper/2603.20164/) (2026)
- [Grounded Vision-Language Interpreter for Long-Horizon Bimanual Task and Motion Planning](https://mlsanity.com/paper/2506.03270/) (2025)
- [Large Language Models for Combinatorial Optimization of Design Structure Matrix](https://mlsanity.com/paper/2506.09749/) (2025)
- [Tactical Decision for Multi-UGV Confrontation with a Vision-Language Model-Based Commander](https://mlsanity.com/paper/2507.11079/) (2025)
- [Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks](https://mlsanity.com/paper/2511.07410/) (2025)
- [What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agents](https://mlsanity.com/paper/2608.03099/) (2026)
- [Safe Task Planning with Long-Term Graph Memory for Embodied Agents](https://mlsanity.com/paper/2609.08444/) (2026)
- [SAGE: Symbolic Action-Gating and Editing for LLM Task Planners](https://mlsanity.com/paper/2609.34268/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However, they fall short in decoding visual cues. LLMs have limited direct perception of the world, which leads to a deficient grasp of the current state of the world. By contrast, the emergence of visual language models (VLMs) fills this gap by integrating visual perception modules, which can enhance the autonomy of robotic task planning. Despite these advancements, VLMs still face challenges, such as the potential for task execution errors, even when provided with accurate instructions. To address such issues, this paper proposes a ReplanVLM framework for robotic task planning. In this study, we focus on error correction interventions. An internal error correction mechanism and an external error correction mechanism are presented to correct errors under corresponding phases. A replan strategy is developed to replan tasks or correct error codes when task execution fails. Experimental results on real robots and in simulation environments have demonstrated the superiority of the proposed framework, with higher success rates and robust error correction capabilities in open-world tasks. Videos of our experiments are available at https://youtu.be/NPk2pWKazJc.