mlsanityData to 7 Oct 2026

ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

arXiv 2206.06994 ↗First posted 14 Jun 2022cs.AI cs.CV cs.ROcited by 369

Not on the map. Why?

Cited by 36 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Massive datasets and high-capacity models have driven many recent advancements in computer vision and natural language understanding. This work presents a platform to enable similar success stories in Embodied AI. We propose ProcTHOR, a framework for procedural generation of Embodied AI environments. ProcTHOR enables us to sample arbitrarily large datasets of diverse, interactive, customizable, and performant virtual environments to train and evaluate embodied agents across navigation, interaction, and manipulation tasks. We demonstrate the power and potential of ProcTHOR via a sample of 10,000 generated houses and a simple neural model. Models trained using only RGB images on ProcTHOR, with no explicit mapping and no human task supervision produce state-of-the-art results across 6 embodied AI benchmarks for navigation, rearrangement, and arm manipulation, including the presently running Habitat 2022, AI2-THOR Rearrangement 2022, and RoboTHOR challenges. We also demonstrate strong 0-shot results on these benchmarks, via pre-training on ProcTHOR with no fine-tuning on the downstream benchmark, often beating previous state-of-the-art systems that access the downstream training data.

Key facts. Cited by 369 papers in the map (data through 7 Oct 2026). Builds on 96 of its 144 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

96 of 144 references.

References · 96

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  12. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
38 more
  1. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  10. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  11. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  12. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  13. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  14. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  15. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  16. 2018On Evaluation of Embodied Navigation Agentscited by 526
  17. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  18. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  19. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  20. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  21. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  22. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  23. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  24. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  25. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  26. 2022Kubric: A scalable dataset generatorcited by 373
  27. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  28. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  29. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  30. 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 313
  31. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  32. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  33. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  34. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  35. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  36. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  37. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  38. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 51; 2024: 68; 2025: 117; 2026: 121.

369 papers.

Sort:

By area:

  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  4. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  5. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  6. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  7. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  8. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  9. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  11. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  12. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
38 more
  1. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  4. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 142
  5. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  6. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  7. 2024Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generationcited by 129
  8. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  9. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  10. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  11. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  12. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  13. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 106
  14. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  15. 2022Navigating to Objects in the Real Worldcited by 104
  16. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  17. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  18. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  19. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 91
  20. 2024I-Design: Personalized LLM Interior Designercited by 87
  21. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  22. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  23. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 81
  24. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  25. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  26. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  27. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  28. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  29. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  30. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  31. 20253D and 4D World Modeling: A Surveycited by 64
  32. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  33. 2023OBJECT 3DIT: Language-guided 3D-aware Image Editingcited by 62
  34. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  35. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  36. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  37. 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
  38. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54

50 most cited of 369 All 369 (JSON)