mlsanityData to 7 Oct 2026

RePLan: Robotic Replanning with Perception and Language Models

arXiv 2401.04157 ↗First posted 8 Jan 2024cs.ROcited by 39

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Advancements in large language models (LLMs) have demonstrated their potential in facilitating high-level reasoning, logical reasoning and robotics planning. Recently, LLMs have also been able to generate reward functions for low-level robot actions, effectively bridging the interface between high-level planning and low-level robot control. However, the challenge remains that even with syntactically correct plans, robots can still fail to achieve their intended goals due to imperfect plans or unexpected environmental issues. To overcome this, Vision Language Models (VLMs) have shown remarkable success in tasks such as visual question answering. Leveraging the capabilities of VLMs, we present a novel framework called Robotic Replanning with Perception and Language Models (RePLan) that enables online replanning capabilities for long-horizon tasks. This framework utilizes the physical grounding provided by a VLM's understanding of the world's state to adapt robot actions when the initial plan fails to achieve the desired goal. We developed a Reasoning and Control (RC) benchmark with eight long-horizon tasks to test our approach. We find that RePLan enables a robot to successfully adapt to unforeseen obstacles while accomplishing open-ended, long-horizon goals, where baseline models cannot, and can be readily applied to real robots. Find more information at https://replan-lm.github.io/replan.github.io/

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. First posted on arXiv 8 Jan 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 12; 2026: 13.

39 papers.

Sort:

By area:

  1. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  2. 2024ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterizationcited by 60
  3. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Modelscited by 46
  4. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  6. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  7. 2024ReplanVLM: Replanning Robotic Tasks with Visual Language Modelscited by 28
  8. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  9. 2024Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfactioncited by 19
  10. 2025Learning Compositional Behaviors from Demonstration and Languagecited by 15
  11. 2025VerifyLLM: LLM-Based Pre-Execution Task Plan Verification for Robotscited by 14
  12. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 11
27 more
  1. 2024Open-World Task and Motion Planning via Vision-Language Model Generated Constraintscited by 9
  2. 2024SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulationcited by 8
  3. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actionscited by 8
  4. 2025REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulationcited by 8
  5. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  6. 2024GameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Gamescited by 7
  7. 2024Creative Problem Solving in Large Language and Vision Models -- What Would it Take?cited by 6
  8. 2026Self-Refining Vision Language Model for Robotic Failure Detection and Reasoningcited by 6
  9. 2024DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipelinecited by 5
  10. 2025STAR: A Foundation Model-driven Framework for Robust Task Planning and Failure Recovery in Robotic Systemscited by 2
  11. 2025RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recoverycited by 2
  12. 2025Grounding Language Models with Semantic Digital Twins for Robotic Planningcited by 2
  13. 2026RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Modelscited by 2
  14. 2026Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulationcited by 2
  15. 2024Long-horizon Embodied Planning with Implicit Logical Inference and Hallucination Mitigationcited by 1
  16. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  17. 2026MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulationcited by 1
  18. 2026The Robot's Inner Critic: Self-Refinement of Social Behaviors through VLM-based Replanningcited by 1
  19. 2026KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysiscited by 1
  20. 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
  21. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarkscited by 0
  22. 2026From Dialogue to Execution: Mixture-of-Agents Assisted Interactive Planning for Behavior Tree-Based Long-Horizon Robot Executioncited by 0
  23. 2026Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planningcited by 0
  24. 2026APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughtscited by 0
  25. 2026Interventional Causal Circuits for Safe Robot Action Testing and Failure Recoverycited by 0
  26. 2026Resilience Matters for Embodied Agents System: New Metrics, Systematic Evaluation, and Optimizationcited by 0
  27. 2026ProTracer: Proprioception-Guided Failure Diagnosis in Robot Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LERa: Replanning with Visual Feedback in Instruction Following
  2. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation
  3. 2025ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learning
  4. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  5. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  6. 2025Reinforced Embodied Planning with Verifiable Reward for Real-World Robotic Manipulation
  7. 2026ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback ↗
  8. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks