mlsanityData to 14 Oct 2026

Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots

arXiv 2310.13724 ↗First posted 19 Oct 2023cs.HC cs.AI cs.CVcited by 228

Not on the map. Why?

Cited by 34 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input. (3) Collaborative tasks: studying two collaborative tasks, Social Navigation and Social Rearrangement. Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene. These contributions allow us to study end-to-end learned and heuristic baselines for human-robot collaboration in-depth, as well as evaluate them with humans in the loop. Our experiments demonstrate that learned robot policies lead to efficient task completion when collaborating with unseen humanoid agents and human partners that might exhibit behaviors that the robot has not seen before. Additionally, we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent. Furthermore, our experiments using the human-in-the-loop tool demonstrate that our automated evaluation with humanoids can provide an indication of the relative ordering of different policies when evaluated with real human collaborators. Habitat 3.0 unlocks interesting new features in simulators for Embodied AI, and we hope it paves the way for a new frontier of embodied human-AI interaction capabilities.

Key facts. Cited by 228 papers in the map (data through 14 Oct 2026). Builds on 76 of its 91 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

76 of 91 references.

References · 76

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  7. 2022Human Motion Diffusion Modelcited by 1,080
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  10. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  11. 2019The StarCraft Multi-Agent Challengecited by 597
  12. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
38 more
  1. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  2. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  3. 2018On Evaluation of Embodied Navigation Agentscited by 547
  4. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  5. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  6. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  7. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  9. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  10. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  12. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  13. 2016Learning Multiagent Communication with Backpropagationcited by 366
  14. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  15. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  16. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  17. 2017Embodied Question Answeringcited by 336
  18. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  19. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  20. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 263
  21. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  22. 2019Vision-and-Dialog Navigationcited by 250
  23. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  24. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  25. 2018Unity: A General Platform for Intelligent Agentscited by 236
  26. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  27. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  28. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  29. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  30. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  31. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 194
  32. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 193
  33. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 185
  34. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 167
  35. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  36. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  37. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  38. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 63; 2025: 77; 2026: 82.

228 papers.

Sort:

By area:

  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  3. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 156
  4. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  5. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  6. 2024Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodologycited by 95
  7. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  8. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  9. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  10. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 71
  11. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 69
  12. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
38 more
  1. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 63
  2. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  3. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  4. 2025Embodied AI Agents: Modeling the Worldcited by 61
  5. 2024GRUtopia: Dream General Robots in a City at Scalecited by 59
  6. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  7. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  8. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 49
  9. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 48
  10. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 48
  11. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 47
  12. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 45
  13. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  14. 2024ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 38
  15. 2025OctoNav: Towards Generalist Embodied Navigationcited by 34
  16. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  17. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  18. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 30
  19. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  20. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 27
  21. 2023Revisit Human-Scene Interaction via Space Occupancycited by 25
  22. 2024An Interactive Agent Foundation Modelcited by 25
  23. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  24. 2025Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loopcited by 25
  25. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24
  26. 2024Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RLcited by 23
  27. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 22
  28. 2024GarmentLab: A Unified Simulation and Benchmark for Garment Manipulationcited by 22
  29. 2024MetaUrban: An Embodied AI Simulation Platform for Urban Micromobilitycited by 21
  30. 2024Neural Fields in Robotics: A Surveycited by 21
  31. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  32. 2025DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Datacited by 21
  33. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  34. 2026Towards Generalizable Robotic Manipulation in Dynamic Environmentscited by 20
  35. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 19
  36. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 19
  37. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 19
  38. 2024InterFusion: Text-Driven Generation of 3D Human-Object Interactioncited by 18

50 most cited of 228 All 228 (JSON)