mlsanityData to 7 Oct 2026

HomeRobot: Open-Vocabulary Mobile Manipulation

arXiv 2306.11565 ↗First posted 20 Jun 2023cs.RO cs.AI cs.CVcited by 104

Not on the map. Why?

Abstract (arXiv)

HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks. Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. This is a foundational challenge for robots to be useful assistants in human environments, because it involves tackling sub-problems from across robotics: perception, language understanding, navigation, and manipulation are all essential to OVMM. In addition, integration of the solutions to these sub-problems poses its own substantial challenges. To drive research in this area, we introduce the HomeRobot OVMM benchmark, where an agent navigates household environments to grasp novel objects and place them on target receptacles. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch to encourage replication of real-world experiments across labs. We implement both reinforcement learning and heuristic (model-based) baselines and show evidence of sim-to-real transfer. Our baselines achieve a 20% success rate in the real world; our experiments identify ways future research work improve performance. See videos on our website: https://ovmm.github.io/.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 76 of its 116 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

76 of 116 references.

References · 76

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Segment Anythingcited by 8,158
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
38 more
  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  2. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  5. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  6. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  7. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  9. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  10. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  12. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  13. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  14. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  15. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  16. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  17. 2020Integrated Task and Motion Planningcited by 281
  18. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  19. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  20. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  21. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  22. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  23. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  24. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  25. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  26. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  27. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  28. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  29. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  30. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  31. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  32. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  33. 2022Navigating to Objects in the Real Worldcited by 104
  34. 2021Visual Room Rearrangementcited by 101
  35. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  36. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 89
  37. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  38. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 38; 2025: 33; 2026: 26.

104 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  5. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  6. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  7. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 65
  8. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  9. 2025Embodied AI Agents: Modeling the Worldcited by 57
  10. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  11. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  12. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
38 more
  1. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  2. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  3. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  4. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  5. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  7. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  8. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  9. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  10. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  11. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  12. 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
  13. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  14. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  15. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  16. 2024Adaptive Mobile Manipulation for Articulated Objects In the Open Worldcited by 30
  17. 2024Prioritized Semantic Learning for Zero-shot Instance Navigationcited by 30
  18. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  19. 2026ABot-M0.5: Unified Mobility-and-Manipulation World Action Modelcited by 28
  20. 2025AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulationcited by 21
  21. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  22. 2023Harmonic Mobile Manipulationcited by 16
  23. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  24. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  25. 2023UniTeam: Open Vocabulary Mobile Manipulation Challengecited by 15
  26. 2024Verifiably Following Complex Robot Instructions with Foundation Modelscited by 15
  27. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
  28. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 15
  29. 2025EchoVLA: Robotic Vision-Language-Action Model with Synergistic Declarative Memory for Mobile Manipulationcited by 15
  30. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Mapscited by 13
  31. 2024LEGENT: Open Platform for Embodied Agentscited by 12
  32. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  33. 2024Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Controlcited by 11
  34. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  35. 2026World Action Models: A Surveycited by 10
  36. 2023Exploitation-Guided Exploration for Semantic Embodied Navigationcited by 9
  37. 2024Seeing the Unseen: Visual Common Sense for Semantic Placementcited by 9
  38. 2024Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challengecited by 9

50 most cited of 104 All 104 (JSON)