mlsanityData to 14 Oct 2026

Gibson Env: Real-World Perception for Embodied Agents

arXiv 1808.10654 ↗First posted 31 Aug 2018cs.AI cs.CV cs.GRcited by 481

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtualizing real spaces, rather than using artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism, "Goggles", enabling deploying the trained models in real-world without needing further domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.

Key facts. Cited by 481 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 1; 2021: 87; 2022: 89; 2023: 76; 2024: 79; 2025: 85; 2026: 62.

481 papers.

Sort:

By area:

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  2. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  3. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  4. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  5. 2022Kubric: A scalable dataset generatorcited by 381
  6. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  7. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  8. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  9. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  11. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  12. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
38 more
  1. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  2. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  3. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  4. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 205
  5. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  6. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  7. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 194
  8. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  9. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  10. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  11. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  12. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 167
  13. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  14. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  15. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  16. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  17. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 141
  18. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  19. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  20. 2023Large Language Models for Robotics: A Surveycited by 133
  21. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  22. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  23. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  24. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  25. 2021Pathdreamer: A World Model for Indoor Navigationcited by 118
  26. 2021Capturing, Reconstructing, and Simulating: the UrbanScene3D Datasetcited by 118
  27. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  28. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  29. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 117
  30. 2022Habitat-Matterport 3D Semantics Datasetcited by 110
  31. 2022Navigating to Objects in the Real Worldcited by 106
  32. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 103
  33. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  34. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 98
  35. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 95
  36. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  37. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  38. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79

50 most cited of 481 All 481 (JSON)