mlsanityData to 7 Oct 2026

3D Generation for Embodied AI and Robotic Simulation: A Survey

arXiv 2604.26509 ↗First posted 29 Apr 2026cs.RO cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Embodied AI and robotic systems increasingly depend on scalable, diverse, and physically grounded 3D content for simulation-based training and real-world deployment. While 3D generative modeling has advanced rapidly, embodied applications impose requirements far beyond visual realism: generated objects must carry kinematic structure and material properties, scenes must support interaction and task execution, and the resulting content must bridge the gap between simulation and reality. This survey reviews 3D generation for embodied AI and organizes the literature around three roles that 3D generation plays in embodied systems. In Data Generator, 3D generation produces simulation-ready objects and assets, including articulated, physically grounded, and deformable content for downstream interaction; in Simulation Environments, it constructs interactive and task-oriented worlds, spanning structure-aware, controllable, and agentic scene generation; and in Sim2Real Bridge, it supports digital twin reconstruction, data augmentation, and synthetic demonstrations for downstream robot learning and real-world transfer. We also show that the field is shifting from visual realism toward interaction readiness, and we identify the main bottlenecks, including limited physical annotations, the gap between geometric quality and physical validity, fragmented evaluation, and the persistent sim-to-real divide, that must be addressed for 3D generation to become a dependable foundation for embodied intelligence. Our project page is at https://3dgen4robot.github.io.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 203 of its 233 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

203 of 233 references.

References · 203

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  11. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  12. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
38 more
  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2018World Modelscited by 1,444
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  8. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  9. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  10. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  13. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  14. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  15. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  16. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  17. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  18. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  19. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  20. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  21. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  22. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  23. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  24. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  25. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  26. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  27. 2018On Evaluation of Embodied Navigation Agentscited by 526
  28. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  29. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  30. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  31. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 449
  32. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  33. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  34. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  35. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  36. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  37. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  38. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308

50 most cited of 203 All 203 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI
  2. 2026EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation
  3. 2026D3D-GEN: Robot-Aware Domain-Grounded Interactive 3D World Generation for Social Robotics ↗
  4. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning
  5. 2025Procedural Generation of Articulated Simulation-Ready Assets
  6. 2026SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining
  7. 2026RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction
  8. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Review