mlsanityData to 13 Oct 2026

Phone2Proc: Bringing Robust Robots Into Our Chaotic World

arXiv 2212.04819 ↗First posted 8 Dec 2022cs.RO cs.AI cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

Training embodied agents in simulation has become mainstream for the embodied AI community. However, these agents often struggle when deployed in the physical world due to their inability to generalize to real-world environments. In this paper, we present Phone2Proc, a method that uses a 10-minute phone scan and conditional procedural generation to create a distribution of training scenes that are semantically similar to the target environment. The generated scenes are conditioned on the wall layout and arrangement of large objects from the scan, while also sampling lighting, clutter, surface textures, and instances of smaller objects with randomized placement and materials. Leveraging just a simple RGB camera, training with Phone2Proc shows massive improvements from 34.7% to 70.7% success rate in sim-to-real ObjectNav performance across a test suite of over 200 trials in diverse real-world environments, including homes, offices, and RoboTHOR. Furthermore, Phone2Proc's diverse distribution of generated scenes makes agents remarkably robust to changes in the real world, such as human movement, object rearrangement, lighting changes, or clutter.

Key facts. Cited by 25 papers in the map (data through 13 Oct 2026). Builds on 57 of its 77 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

57 of 77 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,352
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,411
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,169
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 859
  7. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 583
  8. 2018On Evaluation of Embodied Navigation Agentscited by 531
  9. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  10. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  11. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 371
  12. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 353
38 more
  1. 2020Learning to Explore using Active Neural SLAMcited by 308
  2. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  3. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  4. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  5. 2018Semi-parametric Topological Memory for Navigationcited by 204
  6. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  7. 2020Neural Topological SLAM for Visual Navigationcited by 193
  8. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  9. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  10. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  11. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  12. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 153
  13. 2018Visual Semantic Navigation using Scene Priorscited by 127
  14. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  15. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  16. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  17. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  18. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  19. 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
  20. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  21. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  22. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  23. 2019Deep Visual MPC-Policy Learning for Navigationcited by 54
  24. 2020AllenAct: A Framework for Embodied AI Researchcited by 54
  25. 2021RobustNav: Towards Benchmarking Robustness in Embodied Navigationcited by 52
  26. 2021Learning to Map for Active Semantic Goal Navigationcited by 48
  27. 2021Auto-Tuned Sim-to-Real Transfercited by 45
  28. 2020Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effectscited by 40
  29. 2019PyRobot: An Open-source Robotics Framework for Research and Benchmarkingcited by 39
  30. 2021Planar Robot Casting with Real2Sim2Real Self-Supervised Learningcited by 38
  31. 2019Scaling Local Control to Large-Scale Topological Navigationcited by 37
  32. 2020Bi-directional Domain Adaptation for Sim2Real Transfer of Embodied Navigation Agentscited by 33
  33. 2022Is Mapping Necessary for Realistic PointGoal Navigation?cited by 32
  34. 2022DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objectscited by 29
  35. 2022Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigationcited by 28
  36. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 20
  37. 2022A Real2Sim2Real Method for Robust Object Grasping with Neural Surface Reconstructioncited by 18
  38. 2018Visual Memory for Robust Path Followingcited by 16

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 7; 2026: 6.

25 papers.

Sort:

By area:

  1. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 258
  2. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  3. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 107
  4. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 67
  5. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  6. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  7. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 30
  8. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  9. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 17
  10. 2023Harmonic Mobile Manipulationcited by 16
  11. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  12. 2024GRS: Generating Robotic Simulation Tasks from Real-World Imagescited by 13
13 more
  1. 2024Robot Learning with Super-Linear Scalingcited by 11
  2. 2025EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Devicecited by 11
  3. 2024Seeing the Unseen: Visual Common Sense for Semantic Placementcited by 9
  4. 2024DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenescited by 5
  5. 2025UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videoscited by 5
  6. 2026ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstaclescited by 5
  7. 2024One-shot Video Imitation via Parameterized Symbolic Abstraction Graphscited by 3
  8. 2026From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluationcited by 1
  9. 2025MoK-RAG: Mixture of Knowledge Paths Enhanced Retrieval-Augmented Generation for Embodied AI Environmentscited by 0
  10. 2026GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB imagescited by 0
  11. 2026Semantic Radiance Fields as Simulators for Spatial Reasoning in Real-World Scenescited by 0
  12. 2026LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstructioncited by 0
  13. 2026EnvDreamer: Large-Scale Multimodal-to-Environment Generation for Embodied AIcited by 0