mlsanityData to 14 Oct 2026

RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

arXiv 2311.01455 ↗First posted 2 Nov 2023cs.RO cs.AI cs.CVcited by 218

Not on the map. Why?

Cited by 41 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present RoboGen, a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation. RoboGen leverages the latest advancements in foundation and generative models. Instead of directly using or adapting these models to produce policies or low-level actions, we advocate for a generative scheme, which uses these models to automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision. Our approach equips a robotic agent with a self-guided propose-generate-learn cycle: the agent first proposes interesting tasks and skills to develop, and then generates corresponding simulation environments by populating pertinent objects and assets with proper spatial configurations. Afterwards, the agent decomposes the proposed high-level task into sub-tasks, selects the optimal learning approach (reinforcement learning, motion planning, or trajectory optimization), generates required training supervision, and then learns policies to acquire the proposed skill. Our work attempts to extract the extensive and versatile knowledge embedded in large-scale models and transfer them to the field of robotics. Our fully generative pipeline can be queried repeatedly, producing an endless stream of skill demonstrations associated with diverse tasks and environments.

Key facts. Cited by 218 papers in the map (data through 14 Oct 2026). Builds on 79 of its 148 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

79 of 148 references.

References · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  11. 2019Solving Rubik's Cube with a Robot Handcited by 734
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  13. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  14. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  15. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  16. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  17. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  18. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  19. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  20. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  21. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  22. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  23. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  24. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  25. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  26. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  27. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  28. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  29. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  30. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  31. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  32. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 199
  33. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  34. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  35. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  36. 2021A System for General In-Hand Object Re-Orientationcited by 165
  37. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 159
  38. 2023Robot Parkour Learningcited by 149

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 54; 2025: 68; 2026: 93.

218 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  2. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  3. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  5. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  6. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  7. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  8. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  9. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  10. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  11. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  12. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
38 more
  1. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  2. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 88
  3. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 83
  4. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 83
  5. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
  6. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 67
  7. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 67
  8. 2024GRUtopia: Dream General Robots in a City at Scalecited by 59
  9. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priorscited by 54
  10. 2025RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learningcited by 54
  11. 2026ASPIRE: Agentic /Skills Discovery for Roboticscited by 53
  12. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
  13. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 52
  14. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 50
  15. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  16. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  17. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 46
  18. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  19. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 43
  20. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  21. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  22. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 38
  23. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 38
  24. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 37
  25. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  26. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  27. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 34
  28. 2024LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Contentcited by 33
  29. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  30. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 32
  31. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  32. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 30
  33. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28
  34. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  35. 2024RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulationcited by 26
  36. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 26
  37. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 24
  38. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 22

50 most cited of 218 All 218 (JSON)