3D Generation for Embodied AI and Robotic Simulation: A Survey
Not on the map. Why?
Abstract (arXiv)
Embodied AI and robotic systems increasingly depend on scalable, diverse, and physically grounded 3D content for simulation-based training and real-world deployment. While 3D generative modeling has advanced rapidly, embodied applications impose requirements far beyond visual realism: generated objects must carry kinematic structure and material properties, scenes must support interaction and task execution, and the resulting content must bridge the gap between simulation and reality. This survey reviews 3D generation for embodied AI and organizes the literature around three roles that 3D generation plays in embodied systems. In Data Generator, 3D generation produces simulation-ready objects and assets, including articulated, physically grounded, and deformable content for downstream interaction; in Simulation Environments, it constructs interactive and task-oriented worlds, spanning structure-aware, controllable, and agentic scene generation; and in Sim2Real Bridge, it supports digital twin reconstruction, data augmentation, and synthetic demonstrations for downstream robot learning and real-world transfer. We also show that the field is shifting from visual realism toward interaction readiness, and we identify the main bottlenecks, including limited physical annotations, the gap between geometric quality and physical validity, fragmented evaluation, and the persistent sim-to-real divide, that must be addressed for 3D generation to become a dependable foundation for embodied intelligence. Our project page is at https://3dgen4robot.github.io.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 203 of its 233 references that are in the map. First posted on arXiv 29 Apr 2026.
Builds on
203 of 233 references.
References · 203
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
38 more
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
- 2018World Modelscited by 1,444
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
- 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
- 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
- 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
- 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
- 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
- 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
- 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
- 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
- 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
- 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
- 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
- 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
- 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
- 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
- 2018On Evaluation of Embodied Navigation Agentscited by 526
- 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
- 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
- 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
- 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 449
- 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
- 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
- 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
- 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
- 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
- 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
- 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
50 most cited of 203 All 203 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI
- 2026EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation
- 2026D3D-GEN: Robot-Aware Domain-Grounded Interactive 3D World Generation for Social Robotics ↗
- 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning
- 2025Procedural Generation of Articulated Simulation-Ready Assets
- 2026SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining
- 2026RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction
- 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Review