mlsanityData to 14 Oct 2026

GenSim: Generating Robotic Simulation Tasks via Large Language Models

arXiv 2310.01361 ↗First posted 2 Oct 2023cs.LG cs.CL cs.CVcited by 125

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data to demonstrate significant task-level generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GenSim, has two modes: goal-directed generation, wherein a target task is given to the LLM and the LLM proposes a task curriculum to solve the target task, and exploratory generation, wherein the LLM bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, on which we conduct supervised finetuning and evaluate several LLMs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world and outperform baselines by 25%. See the project website (https://liruiw.github.io/gensim) for code, demos, and videos.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 41 of its 57 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

41 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Text2Motion: From Natural Language Instructions to Feasible Plans21 Mar
  2. 2019Solving Rubik's Cube with a Robot Hand16 Oct

Other references · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
27 more
  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  7. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  9. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  10. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  11. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  12. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  13. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  14. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  15. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  16. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  17. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 175
  18. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 159
  19. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  20. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 93
  21. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 85
  22. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 84
  23. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 70
  24. 2023Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Promptingcited by 46
  25. 2023How Can Large Language Models Help Humans in Design and Manufacturing?cited by 41
  26. 2020Adaptive Procedural Task Generation for Hard-Exploration Problemscited by 20
  27. 2023LaMPP: Language Models as Probabilistic Priors for Perception and Actioncited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 33; 2025: 44; 2026: 45.

125 papers.

Sort:

By area:

  1. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  2. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 276
  3. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  4. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  5. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 125
  8. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
  9. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  10. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 83
  11. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 83
  12. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
38 more
  1. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 79
  2. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  3. 2024GRUtopia: Dream General Robots in a City at Scalecited by 59
  4. 2025RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learningcited by 54
  5. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
  6. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
  7. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  8. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  9. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 39
  10. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 38
  11. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  12. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 34
  13. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 26
  14. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 24
  15. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 22
  16. 2025VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotioncited by 22
  17. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 20
  18. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 20
  19. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  20. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 18
  21. 2024Eurekaverse: Environment Curriculum Generation via Large Language Modelscited by 16
  22. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 15
  23. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetcited by 15
  24. 2024GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitationcited by 14
  25. 2024GRS: Generating Robotic Simulation Tasks from Real-World Imagescited by 13
  26. 2025Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixelscited by 13
  27. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  28. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 12
  29. 2024Robot Learning with Super-Linear Scalingcited by 11
  30. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulationcited by 11
  31. 2025Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planningcited by 11
  32. 2024FactorSim: Generative Simulation via Factorized Representationcited by 10
  33. 2025Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generationcited by 10
  34. 2026PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinementcited by 10
  35. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 10
  36. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 10
  37. 2025Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learningcited by 9
  38. 2025Steerable Scene Generation with Post Training and Inference-Time Searchcited by 9

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models
  2. 2024On the Evaluation of Generative Robotic Simulations
  3. 2026ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation
  4. 2025LLM Trainer: Automated Robotic Data Generation via Demonstration Augmentation using LLMs
  5. 2026Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
  6. 2025InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy
  7. 2026GenPHRI: Agentic Generative Simulation for Physical Human-Robot Interaction
  8. 2024Grounding Large Language Models In Embodied Environment With Imperfect World Models