mlsanityData to 7 Oct 2026

GRS: Generating Robotic Simulation Tasks from Real-World Images

arXiv 2410.15536 ↗First posted 20 Oct 2024cs.RO cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

We introduce GRS (Generating Robotic Simulation tasks), a system addressing real-to-sim for robotic simulations. GRS creates digital twin simulations from single RGB-D observations with solvable tasks for virtual agent training. Using vision-language models (VLMs), our pipeline operates in three stages: 1) scene comprehension with SAM2 for segmentation and object description, 2) matching objects with simulation-ready assets, and 3) generating appropriate tasks. We ensure simulation-task alignment through generated test suites and introduce a router that iteratively refines both simulation and test code. Experiments demonstrate our system's effectiveness in object correspondence and task environment generation through our novel router mechanism.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 64 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

33 of 64 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  5. 2024Genie: Generative Interactive Environmentscited by 706
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  7. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  8. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  9. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  10. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  11. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  12. 2024Diffusion Models Are Real-Time Game Enginescited by 282
21 more
  1. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  2. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  3. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  4. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  5. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  6. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  7. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 85
  8. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  9. 2017Learning to Infer Graphics Programs from Hand-Drawn Imagescited by 73
  10. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 59
  11. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databasescited by 43
  12. 2024Physically Compatible 3D Object Modeling from a Single Imagecited by 38
  13. 2022Phone2Proc: Bringing Robust Robots Into Our Chaotic Worldcited by 24
  14. 2024RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulationcited by 24
  15. 2023Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Modelscited by 22
  16. 2024Re-Thinking Inverse Graphics With Large Language Modelscited by 20
  17. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 19
  18. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  19. 2024SceneMotifCoder: Example-driven Visual Program Learning for Generating 3D Object Arrangementscited by 16
  20. 2018Visual Robot Task Planningcited by 11
  21. 2024FactorSim: Generative Simulation via Factorized Representationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 6.

13 papers.

Sort:

By area:

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  2. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 19
  3. 2025Robot Learning from a Physical World Modelcited by 18
  4. 2024Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modelingcited by 10
  5. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  6. 2025Robot Learning from Any Imagescited by 7
  7. 2026Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agentscited by 6
  8. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 3
  9. 2025TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulationcited by 2
  10. 2025IGen: Scalable Data Generation for Robot Learning from Open-World Imagescited by 1
  11. 2026RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generationcited by 1
  12. 2026GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learningcited by 1
1 more
  1. 2026DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Evaluation of Generative Robotic Simulations
  2. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs
  3. 2026Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
  4. 2026Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation
  5. 2026Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
  6. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
  7. 2026SR-Platform: An Agentic Pipeline for Natural Language-Driven Robot Simulation Environment Synthesis
  8. 2026ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation