mlsanityData to 7 Oct 2026

GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs

arXiv 2410.03645 ↗First posted 4 Oct 2024cs.RO cs.AI cs.CVcited by 48

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Robotic simulation today remains challenging to scale up due to the human efforts required to create diverse simulation tasks and scenes. Simulation-trained policies also face scalability issues as many sim-to-real methods focus on a single task. To address these challenges, this work proposes GenSim2, a scalable framework that leverages coding LLMs with multi-modal and reasoning capabilities for complex and realistic simulation task creation, including long-horizon tasks with articulated objects. To automatically generate demonstration data for these tasks at scale, we propose planning and RL solvers that generalize within object categories. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. To utilize such data, we propose an effective multi-task language-conditioned policy architecture, dubbed proprioceptive point-cloud transformer (PPT), that learns from the generated demonstrations and exhibits strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the policy architecture, we show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data, which enhances the policy performance by 20% compared with training exclusively on limited real data.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 57 of its 74 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

57 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Models2 Oct
  2. 2023Text2Motion: From Natural Language Instructions to Feasible Plans21 Mar
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  4. 2020SAPIEN: A SimulAted Part-based Interactive ENvironment19 Mar
  5. 2019Solving Rubik's Cube with a Robot Hand16 Oct

Other references · 52

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  8. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  9. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
38 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2024Genie: Generative Interactive Environmentscited by 706
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  7. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  11. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  13. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  14. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  15. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  16. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  17. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  18. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  19. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  20. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  21. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  22. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  23. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  24. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  25. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  26. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  27. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  28. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  29. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  30. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 103
  31. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  32. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  33. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 78
  34. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 71
  35. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  36. 2023How Can Large Language Models Help Humans in Design and Manufacturing?cited by 41
  37. 2020Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Cloudscited by 24
  38. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 22

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 17; 2026: 28.

48 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  3. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  4. 2024Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimationcited by 73
  5. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 48
  6. 2025InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policycited by 46
  7. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  8. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  9. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 18
  10. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  11. 2025Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robotscited by 11
  12. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulationcited by 10
36 more
  1. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 9
  2. 2025AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learningcited by 8
  3. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 7
  4. 2026Beyond Short-Horizon: VQ-Memory for Robust Long-Horizon Manipulation in Non-Markovian Simulation Benchmarkscited by 5
  5. 2025IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Modelscited by 4
  6. 2025D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentationcited by 4
  7. 2025R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulationcited by 4
  8. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  9. 2025EvoCurr: Self-evolving Curriculum with Behavior Code Generation for Complex Decision-makingcited by 3
  10. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Modelscited by 3
  11. 2026From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Datacited by 3
  12. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applicationscited by 3
  13. 2026MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalizationcited by 3
  14. 2026AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondencecited by 2
  15. 2026EmbodiedSWE: Coding Agents for Long Horizon Dexterous Roboticscited by 2
  16. 2024On the Evaluation of Generative Robotic Simulationscited by 1
  17. 2025dARt Vinci: Egocentric Data Collection for Surgical Robot Learning at Scalecited by 1
  18. 2026RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generationcited by 1
  19. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  20. 2026SADP: Subgoal-Aware Diffusion Policy for Long-Horizon Manipulation Learned from Foundation Model Generated Demonstrationscited by 1
  21. 2026Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agentscited by 1
  22. 2025Look Before You Leap: Using Serialized State Machine for Language Conditioned Robotic Manipulationcited by 0
  23. 2025BLAZER: Bootstrapping LLM-based Manipulation Agents with Zero-Shot Data Generationcited by 0
  24. 2025Sceniris: A Fast Procedural Scene Generation Frameworkcited by 0
  25. 2026V-CAGE: Context-Aware Generation and Verification for Scalable Long-Horizon Embodied Taskscited by 0
  26. 2026GenPHRI: Agentic Generative Simulation for Physical Human-Robot Interactioncited by 0
  27. 2026AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulationcited by 0
  28. 20263D Generation for Embodied AI and Robotic Simulation: A Surveycited by 0
  29. 2026R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policiescited by 0
  30. 2026PRISM: Personalized Robotic Dataset Generation via Image-based Scene and Motion Synthesiscited by 0
  31. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0
  32. 2026SUN: Agentic Robot Policy Learning with Persistent Task Programscited by 0
  33. 2026RoboCousin: Build Your Own Simulation Playground for Robust Bimanual Robotic Manipulationcited by 0
  34. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  35. 2026Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Constructioncited by 0
  36. 2026Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models
  2. 2026ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation
  3. 2026Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
  4. 2025Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware
  5. 2025LLM Trainer: Automated Robotic Data Generation via Demonstration Augmentation using LLMs
  6. 2026Point Bridge: 3D Representations for Cross Domain Policy Learning
  7. 2026MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation
  8. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Models