mlsanityData to 7 Oct 2026

Building Generalizable Agents with a Realistic and Rich 3D Environment

arXiv 1801.02209 ↗First posted 7 Jan 2018cs.LG cs.AIcited by 109

Not on the map. Why?

Abstract (arXiv)

Teaching an agent to navigate in an unseen 3D environment is a challenging task, even in the event of simulated environments. To generalize to unseen environments, an agent needs to be robust to low-level variations (e.g. color, texture, object changes), and also high-level variations (e.g. layout changes of the environment). To improve overall generalization, all types of variations in the environment have to be taken under consideration via different level of data augmentation steps. To this end, we propose House3D, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset (Song et.al.). The diversity in House3D opens the door towards scene-level augmentation, while the label-rich nature of House3D enables us to inject pixel- & task-level augmentations such as domain randomization (Toubin et. al.) and multi-task training. Using a subset of houses in House3D, we show that reinforcement learning agents trained with an enhancement of different levels of augmentations perform much better in unseen environments than our baselines with raw RGB input by over 8% in terms of navigation success rate. House3D is publicly available at http://github.com/facebookresearch/House3D.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 1; 2021: 34; 2022: 21; 2023: 16; 2024: 20; 2025: 10; 2026: 5.

109 papers.

Sort:

By area:

  1. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  3. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  4. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  5. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  6. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  7. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  8. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  9. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  10. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  11. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  12. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
38 more
  1. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  2. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  3. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  4. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  5. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  6. 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordinationcited by 48
  7. 2021Learning to Map for Active Semantic Goal Navigationcited by 47
  8. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
  9. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  10. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  11. 2024HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environmentscited by 32
  12. 20213DB: A Framework for Debugging Computer Vision Modelscited by 31
  13. 2021Deep Learning Based 3D Segmentation: A Surveycited by 30
  14. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  15. 2024Prioritized Semantic Learning for Zero-shot Instance Navigationcited by 30
  16. 2022Cross-View Cross-Scene Multi-View Crowd Countingcited by 28
  17. 2021The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigationcited by 27
  18. 2022Towards Versatile Embodied Navigationcited by 26
  19. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 25
  20. 2024Make-A-Shape: a Ten-Million-scale 3D Shape Modelcited by 25
  21. 2024Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructionscited by 25
  22. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  23. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  24. 2022Ground then Navigate: Language-guided Navigation in Dynamic Scenescited by 20
  25. 2021Collaborative Visual Navigationcited by 18
  26. 2021Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learningcited by 18
  27. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  28. 2023EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervisioncited by 18
  29. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 18
  30. 2021A Systematic Survey of Text Worlds as Embodied Natural Language Environmentscited by 17
  31. 2022Unbiased Directed Object Attention Graph for Object Navigationcited by 17
  32. 2021Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigationcited by 15
  33. 2022Navigating to Objects in Unseen Environments by Distance Predictioncited by 15
  34. 2022HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handoverscited by 15
  35. 2020Towards Target-Driven Visual Navigation in Indoor Scenes via Generative Imitation Learningcited by 14
  36. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  37. 2025IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarioscited by 12
  38. 2022On the Learning Mechanisms in Physical Reasoningcited by 11

50 most cited of 109 All 109 (JSON)