mlsanityData to 7 Oct 2026

DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment

arXiv 2307.00329 ↗First posted 1 Jul 2023cs.RO cs.AIcited by 58

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Large language models (LLMs) encode a vast amount of semantic knowledge and possess remarkable understanding and reasoning capabilities. Previous work has explored how to ground LLMs in robotic tasks to generate feasible and executable textual plans. However, low-level execution in the physical world may deviate from the high-level textual plan due to environmental perturbations or imperfect controller design. In this paper, we propose DoReMi, a novel language model grounding framework that enables immediate Detection and Recovery from Misalignments between plan and execution. Specifically, we leverage LLMs to play a dual role, aiding not only in high-level planning but also generating constraints that can indicate misalignment during execution. Then vision language models (VLMs) are utilized to detect constraint violations continuously. Our pipeline can monitor the low-level execution and enable timely recovery if certain plan-execution misalignment occurs. Experiments on various complex tasks including robot arms and humanoid robots demonstrate that our method can lead to higher task success rates and shorter task completion times. Videos of DoReMi are available at urlhttps://sites.google.com/view/doremi-paper.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 62 references that are in the map. First posted on arXiv 1 Jul 2023.

Builds on

44 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
31 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  5. 2022A Generalist Agentcited by 802
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  13. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  14. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  15. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  16. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  17. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  18. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  19. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  20. 2023Vision-Language Models as Success Detectorscited by 138
  21. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  22. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  23. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  24. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  25. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  26. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  27. 2018Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systemscited by 40
  28. 2021kPAM 2.0: Feedback Control for Category-Level Robotic Manipulationcited by 38
  29. 2022Combining Learning-based Locomotion Policy with Model-based Manipulation for Legged Mobile Manipulatorscited by 32
  30. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  31. 2023Grounding Classical Task Planners via Vision-Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 17; 2025: 15; 2026: 20.

58 papers.

Sort:

By area:

  1. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  2. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  3. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  4. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  5. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  6. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Modelscited by 46
  7. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  8. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  9. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  10. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  11. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  12. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
38 more
  1. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24
  2. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
  3. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  4. 2024Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfactioncited by 19
  5. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  6. 2023LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulationcited by 16
  7. 2024Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Plannerscited by 15
  8. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14
  9. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  10. 2026LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policiescited by 12
  11. 2024Open-World Task and Motion Planning via Vision-Language Model Generated Constraintscited by 9
  12. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  13. 2025MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planningcited by 8
  14. 2026Towards the Harness of Embodied Agentscited by 7
  15. 2023Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recoverycited by 6
  16. 2025Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skillscited by 6
  17. 2026Self-Refining Vision Language Model for Robotic Failure Detection and Reasoningcited by 6
  18. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 6
  19. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Modelscited by 5
  20. 2026VLCP: Vision Language Control Policy Closed-Loop Code Replanning for Robot Manipulationcited by 5
  21. 2024Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modelingcited by 4
  22. 2025A Unified Framework for Real-Time Failure Handling in Robotics Using Vision-Language Models, Reactive Planner and Behavior Treescited by 4
  23. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  24. 2025Robotic Long-Horizon Manipulation with Progressive In-Context Code Generation and Episodic Feedbackcited by 3
  25. 2025ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learningcited by 3
  26. 2024Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skillscited by 2
  27. 2025RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recoverycited by 2
  28. 2026ContextFlow: Hierarchical Task-State Alignment for Long-Horizon Embodied Agentscited by 2
  29. 2025SDA-PLANNER: State-Dependency Aware Adaptive Planner for Embodied Task Planningcited by 1
  30. 2026ThermoAct:Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Makingcited by 1
  31. 2026RoboBRIDGE: A Modular Framework for Bridging Policies to Robust Real-World Robotic Agentscited by 1
  32. 2026TROVE: Adaptive Agent Skill Orchestration via Trace-Grounded Route Validation and Editingcited by 1
  33. 2025Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided and Self-Consistent MLLMs for Task Planning in Instruction-Following Manipulationcited by 0
  34. 2025LangPert: Detecting and Handling Task-level Perturbations for Robust Object Rearrangementcited by 0
  35. 2025FrankenBot: Brain-Morphic Modular Orchestration for Robotic Manipulation with Vision-Language Modelscited by 0
  36. 2025BLAZER: Bootstrapping LLM-based Manipulation Agents with Zero-Shot Data Generationcited by 0
  37. 2026LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratorycited by 0
  38. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  2. 2026ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models
  3. 2025Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
  4. 2026RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
  5. 2026Language Movement Primitives: Grounding Language Models in Robot Motion
  6. 2026Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation ↗
  7. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models
  8. 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification