mlsanityData to 7 Oct 2026

Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V

arXiv 2404.10220 ↗First posted 16 Apr 2024cs.RO cs.AI cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. In this work, we present COME-robot, the first closed-loop robotic system utilizing the GPT-4V vision-language foundation model for open-ended reasoning and adaptive planning in real-world scenarios.COME-robot incorporates two key innovative modules: (i) a multi-level open-vocabulary perception and situated reasoning module that enables effective exploration of the 3D environment and target object identification using commonsense knowledge and situated information, and (ii) an iterative closed-loop feedback and restoration mechanism that verifies task feasibility, monitors execution success, and traces failure causes across different modules for robust failure recovery. Through comprehensive experiments involving 8 challenging real-world mobile and tabletop manipulation tasks, COME-robot demonstrates a significant improvement in task success rate (~35%) compared to state-of-the-art methods. We further conduct comprehensive analyses to elucidate how COME-robot's design facilitates failure recovery, free-form instruction following, and long-horizon task planning.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 38 of its 65 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

38 of 65 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  12. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
26 more
  1. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  2. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  3. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  4. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  5. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 212
  6. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  7. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  8. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  9. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  10. 2023Large Language Models for Robotics: A Surveycited by 132
  11. 2022Navigating to Objects in the Real Worldcited by 104
  12. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  13. 2023GOAT: GO to Any Thingcited by 89
  14. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 86
  15. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  16. 2024RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operationcited by 62
  17. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 60
  18. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  19. 2022Multi-skill Mobile Manipulation for Object Rearrangementcited by 46
  20. 2024RePLan: Robotic Replanning with Perception and Language Modelscited by 39
  21. 2021Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordancescited by 38
  22. 2023ASC: Adaptive Skill Coordination for Robotic Mobile Manipulationcited by 38
  23. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  24. 2018Artificial Intelligence for Long-Term Robot Autonomy: A Surveycited by 21
  25. 2021Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignmentscited by 15
  26. 2021Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspectivecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 22; 2026: 6.

40 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  4. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 31
  7. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 30
  8. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  9. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14
  10. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  11. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  12. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Taskscited by 9
28 more
  1. 2025MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoningcited by 9
  2. 2025ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Taskscited by 9
  3. 2024DISCO: Language-Guided Manipulation with Diffusion Policies and Constrained Inpaintingcited by 8
  4. 2024ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Executioncited by 8
  5. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 8
  6. 2026MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon taskscited by 8
  7. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  8. 2024AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robotscited by 7
  9. 2024WildLMa: Long Horizon Loco-Manipulation in the Wildcited by 6
  10. 2025Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skillscited by 6
  11. 2026See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulationcited by 6
  12. 2024M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenescited by 5
  13. 2024DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipelinecited by 5
  14. 2025Learning Human-Humanoid Coordination for Collaborative Object Carryingcited by 4
  15. 2025OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesiscited by 3
  16. 2026ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Modelscited by 3
  17. 2025LightPlanner: Unleashing the Reasoning Capabilities of Lightweight Large Language Models in Task Planningcited by 2
  18. 2026From Instruction to Event: Sound-Triggered Mobile Manipulationcited by 2
  19. 2024Long-horizon Embodied Planning with Implicit Logical Inference and Hallucination Mitigationcited by 1
  20. 2025RoboAct-CLIP: Video-Driven Pre-training of Atomic Action Understanding for Roboticscited by 1
  21. 2025Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Modelscited by 1
  22. 2025A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real Worldcited by 1
  23. 2025From Code to Action: Hierarchical Learning of Diffusion-VLM Policiescited by 1
  24. 2024RoboGolf: Mastering Real-World Minigolf with a Reflective Multi-Modality Vision-Language Modelcited by 0
  25. 2025Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulationcited by 0
  26. 2025BINDER: Instantly Adaptive Mobile Manipulation with Open-Vocabulary Commandscited by 0
  27. 2026Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inferencecited by 0
  28. 2026FRAMES: Failure Recovery And Monitoring of Embodied Skills for Humanoid Loco-Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps
  2. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics
  3. 2025RoboPilot: Generalizable Dynamic Robotic Manipulation with Dual-thinking Modes
  4. 2026RMMBench: A Comprehensive Benchmark for Robotic Mobile Manipulation
  5. 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation
  6. 2026ANCHOR: A Physically Grounded Closed-Loop Framework for Robust Home-Service Mobile Manipulation ↗
  7. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
  8. 2026UniPlan: Vision-Language Task Planning for Mobile Manipulation with Unified PDDL Formulation