mlsanityData to 7 Oct 2026

Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models

arXiv 2310.18308 ↗First posted 27 Oct 2023cs.RO cs.AI cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Generalist robot manipulators need to learn a wide variety of manipulation skills across diverse environments. Current robot training pipelines rely on humans to provide kinesthetic demonstrations or to program simulation environments and to code up reward functions for reinforcement learning. Such human involvement is an important bottleneck towards scaling up robot learning across diverse tasks and environments. We propose Generation to Simulation (Gen2Sim), a method for scaling up robot skill learning in simulation by automating generation of 3D assets, task descriptions, task decompositions and reward functions using large pre-trained generative models of language and vision. We generate 3D assets for simulation by lifting open-world 2D object-centric images to 3D using image diffusion models and querying LLMs to determine plausible physics parameters. Given URDF files of generated and human-developed assets, we chain-of-thought prompt LLMs to map these to relevant task descriptions, temporal decompositions, and corresponding python reward functions for reinforcement learning. We show Gen2Sim succeeds in learning policies for diverse long horizon tasks, where reinforcement learning with non temporally decomposed reward functions fails. Gen2Sim provides a viable path for scaling up reinforcement learning for robot manipulators in simulation, both by diversifying and expanding task and environment development, and by facilitating the discovery of reinforcement-learned behaviors through temporal task decomposition in RL. Our work contributes hundreds of simulated assets, tasks and demonstrations, taking a step towards fully autonomous robotic manipulation skill acquisition in simulation.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 69 of its 85 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

69 of 85 references.

References · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2016OpenAI Gymcited by 1,684
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  13. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  14. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  15. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  16. 2019Solving Rubik's Cube with a Robot Handcited by 699
  17. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  18. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  19. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  20. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  21. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  22. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  23. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  24. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  25. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  26. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  27. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  28. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  29. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  30. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  31. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  32. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  33. 2021A System for General In-Hand Object Re-Orientationcited by 155
  34. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 135
  35. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 118
  36. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  37. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  38. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 77

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 27; 2026: 18.

62 papers.

Sort:

By area:

  1. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  4. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  5. 2024Real2Code: Reconstruct Articulated Objects via Code Generationcited by 64
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  7. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 41
  8. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  9. 2025EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligencecited by 32
  10. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 29
  11. 2024An Interactive Agent Foundation Modelcited by 25
  12. 2024Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomaliescited by 19
38 more
  1. 2024Neural MP: A Generalist Neural Motion Plannercited by 18
  2. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  3. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  4. 2024Generative AI for Unmanned Vehicle Swarms: Challenges, Applications and Opportunitiescited by 14
  5. 2025EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfercited by 12
  6. 2025Interpretable Generative Models through Post-hoc Concept Bottleneckscited by 11
  7. 2024FactorSim: Generative Simulation via Factorized Representationcited by 9
  8. 2025Steerable Scene Generation with Post Training and Inference-Time Searchcited by 8
  9. 2025AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learningcited by 8
  10. 2023TPA3D: Triplane Attention for Fast Text-to-3D Generationcited by 6
  11. 2025Is Single-View Mesh Reconstruction Ready for Robotics?cited by 6
  12. 2026Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agentscited by 6
  13. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 5
  14. 2025Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applicationscited by 5
  15. 2025MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layoutscited by 5
  16. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  17. 2025D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentationcited by 4
  18. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  19. 2025R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulationcited by 4
  20. 2026Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulationcited by 4
  21. 2026EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AIcited by 4
  22. 2024AnyBipe: An End-to-End Framework for Training and Deploying Bipedal Robots Guided by Large Language Modelscited by 3
  23. 2025Real-Time Verification of Embodied Reasoning for Generative Skill Acquisitioncited by 3
  24. 2025Zero-Shot Visual Generalization in Robot Manipulationcited by 3
  25. 2025Bootstrapping Imitation Learning for Long-horizon Manipulation via Hierarchical Data Collection Spacecited by 3
  26. 2025SLAP: Shortcut Learning for Abstract Planningcited by 3
  27. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 3
  28. 2026i-PhysGaussian: Implicit Physical Simulation for 3D Gaussian Splattingcited by 3
  29. 2024Position Paper: Agent AI Towards a Holistic Intelligencecited by 2
  30. 2024DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Modelcited by 2
  31. 2024Reinforcement Learning via Auxiliary Task Distillationcited by 2
  32. 2025A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Roboticscited by 2
  33. 2025Position: Intelligent Science Laboratory Requires the Integration of Cognitive and Embodied AIcited by 2
  34. 2025Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Searchcited by 2
  35. 2024Deep Dive into Model-free Reinforcement Learning for Biological and Robotic Systems: Theory and Practicecited by 1
  36. 2026RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domainscited by 1
  37. 2026GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learningcited by 1
  38. 2025ConsistentDreamer: View-Consistent Meshes Through Balanced Multi-View Gaussian Optimizationcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Models
  2. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
  3. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos
  4. 2024On the Evaluation of Generative Robotic Simulations
  5. 2026Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
  6. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations
  7. 2026Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
  8. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning