# RePLan: Robotic Replanning with Perception and Language Models

arXiv: https://arxiv.org/abs/2401.04157 · HTML: https://mlsanity.com/paper/2401.04157/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- First posted on arXiv 8 Jan 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Built on by (most cited first, 30 of 41)

- [Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress](https://mlsanity.com/paper/2410.04640/) (2024)
- [ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization](https://mlsanity.com/paper/2401.06949/) (2024)
- [Guiding Long-Horizon Task and Motion Planning with Vision Language Models](https://mlsanity.com/paper/2410.02193/) (2024)
- [Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V](https://mlsanity.com/paper/2404.10220/) (2024)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)
- [ReplanVLM: Replanning Robotic Tasks with Visual Language Models](https://mlsanity.com/paper/2407.21762/) (2024)
- [Robix: A Unified Model for Robot Interaction, Reasoning and Planning](https://mlsanity.com/paper/2509.01106/) (2025)
- [Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction](https://mlsanity.com/paper/2406.05572/) (2024)
- [Learning Compositional Behaviors from Demonstration and Language](https://mlsanity.com/paper/2505.21981/) (2025)
- [VerifyLLM: LLM-Based Pre-Execution Task Plan Verification for Robots](https://mlsanity.com/paper/2507.05118/) (2025)
- [I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models](https://mlsanity.com/paper/2509.16072/) (2025)
- [SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation](https://mlsanity.com/paper/2410.18065/) (2024)
- [Open-World Task and Motion Planning via Vision-Language Model Generated Constraints](https://mlsanity.com/paper/2411.08253/) (2024)
- [Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models](https://mlsanity.com/paper/2512.01946/) (2025)
- [VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions](https://mlsanity.com/paper/2410.20927/) (2024)
- [REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2503.22122/) (2025)
- [Creative Problem Solving in Large Language and Vision Models -- What Would it Take?](https://mlsanity.com/paper/2405.01453/) (2024)
- [GameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Games](https://mlsanity.com/paper/2405.13751/) (2024)
- [Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning](https://mlsanity.com/paper/2602.12405/) (2026)
- [DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline](https://mlsanity.com/paper/2412.01663/) (2024)
- [Long-horizon Embodied Planning with Implicit Logical Inference and Hallucination Mitigation](https://mlsanity.com/paper/2409.15658/) (2024)
- [STAR: A Foundation Model-driven Framework for Robust Task Planning and Failure Recovery in Robotic Systems](https://mlsanity.com/paper/2503.06060/) (2025)
- [RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recovery](https://mlsanity.com/paper/2503.17703/) (2025)
- [Grounding Language Models with Semantic Digital Twins for Robotic Planning](https://mlsanity.com/paper/2506.16493/) (2025)
- [RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models](https://mlsanity.com/paper/2605.09410/) (2026)
- [Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation](https://mlsanity.com/paper/2607.13653/) (2026)
- [Human Centric General Physical Intelligence for Agile Manufacturing Automation](https://mlsanity.com/paper/2508.11960/) (2025)
- [MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation](https://mlsanity.com/paper/2602.16898/) (2026)
- [The Robot's Inner Critic: Self-Refinement of Social Behaviors through VLM-based Replanning](https://mlsanity.com/paper/2603.20164/) (2026)

## Abstract (arXiv)

Advancements in large language models (LLMs) have demonstrated their potential in facilitating high-level reasoning, logical reasoning and robotics planning. Recently, LLMs have also been able to generate reward functions for low-level robot actions, effectively bridging the interface between high-level planning and low-level robot control. However, the challenge remains that even with syntactically correct plans, robots can still fail to achieve their intended goals due to imperfect plans or unexpected environmental issues. To overcome this, Vision Language Models (VLMs) have shown remarkable success in tasks such as visual question answering. Leveraging the capabilities of VLMs, we present a novel framework called Robotic Replanning with Perception and Language Models (RePLan) that enables online replanning capabilities for long-horizon tasks. This framework utilizes the physical grounding provided by a VLM's understanding of the world's state to adapt robot actions when the initial plan fails to achieve the desired goal. We developed a Reasoning and Control (RC) benchmark with eight long-horizon tasks to test our approach. We find that RePLan enables a robot to successfully adapt to unforeseen obstacles while accomplishing open-ended, long-horizon goals, where baseline models cannot, and can be readily applied to real robots. Find more information at https://replan-lm.github.io/replan.github.io/