mlsanityData to 14 Oct 2026

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

arXiv 2106.14405 ↗First posted 28 Jun 2021cs.LG cs.ROcited by 511

Not on the map. Why?

Cited by 36 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack - data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850x real-time) on an 8-GPU node, representing 100x speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies.

Key facts. Cited by 511 papers in the map (data through 14 Oct 2026). Builds on 67 of its 124 references that are in the map. First posted on arXiv 28 Jun 2021.

Builds on

67 of 124 references.

References · 67

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2018Sanity Checks for Saliency Mapscited by 1,156
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2018DeepMind Control Suitecited by 956
  8. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
38 more
  1. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  3. 2018On Evaluation of Embodied Navigation Agentscited by 547
  4. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  5. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  6. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  7. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  8. 2020Integrated Task and Motion Planningcited by 290
  9. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  10. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  11. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 242
  12. 2020Rearrangement: A Challenge for Embodied AIcited by 155
  13. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 155
  14. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  15. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  16. 2013Data-Driven Grasp Synthesis - A Surveycited by 106
  17. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 106
  18. 2021Visual Room Rearrangementcited by 102
  19. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  20. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  21. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  22. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 90
  23. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 89
  24. 2017Learning Sampling Distributions for Robot Motion Planningcited by 82
  25. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 78
  26. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  27. 2020OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasetscited by 74
  28. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  29. 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 69
  30. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  31. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 65
  32. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 61
  33. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  34. 2020Auxiliary Tasks Speed Up Learning PointGoal Navigationcited by 54
  35. 2019Combining Optimal Control and Learning for Visual Navigation in Novel Environmentscited by 52
  36. 2014Informed RRT*: Optimal Sampling-based Path Planning Focused via Direct Sampling of an Admissible Ellipsoidal Heuristiccited by 50
  37. 2017Continuous-Time Gaussian Process Motion Planning via Probabilistic Inferencecited by 49
  38. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 49

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 65; 2023: 110; 2024: 100; 2025: 127; 2026: 92.

511 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  3. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  4. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  5. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  6. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  7. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  8. 2022Kubric: A scalable dataset generatorcited by 381
  9. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  10. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  11. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  12. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
38 more
  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  2. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  3. 2025SAM 3D: 3Dfy Anything in Imagescited by 247
  4. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  5. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  6. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  7. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  8. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  9. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 194
  10. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  11. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  12. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  13. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 167
  14. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
  15. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  16. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  17. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 156
  18. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  19. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 148
  20. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 147
  21. 2024Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generationcited by 137
  22. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  23. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 125
  24. 2023PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objectscited by 118
  25. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  26. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  27. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 116
  28. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  29. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
  30. 2022Habitat-Matterport 3D Semantics Datasetcited by 110
  31. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 109
  32. 2023Controllable Human-Object Interaction Synthesiscited by 109
  33. 2022Navigating to Objects in the Real Worldcited by 106
  34. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  35. 2023Human Motion Generation: A Surveycited by 96
  36. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 95
  37. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  38. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 94

50 most cited of 511 All 511 (JSON)