mlsanityData to 7 Oct 2026

RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

arXiv 2411.11839 ↗First posted 18 Nov 2024cs.RO cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

Efficient acquisition of real-world embodied data has been increasingly critical. However, large-scale demonstrations captured by remote operation tend to take extremely high costs and fail to scale up the data size in an efficient manner. Sampling the episodes under a simulated environment is a promising way for large-scale collection while existing simulators fail to high-fidelity modeling on texture and physics. To address these limitations, we introduce the RoboGSim, a real2sim2real robotic simulator, powered by 3D Gaussian Splatting and the physics engine. RoboGSim mainly includes four parts: Gaussian Reconstructor, Digital Twins Builder, Scene Composer, and Interactive Engine. It can synthesize the simulated data with novel views, objects, trajectories, and scenes. RoboGSim also provides an online, reproducible, and safe evaluation for different manipulation policies. The real2sim and sim2real transfer experiments show a high consistency in the texture and physics. We compared the test results of RoboGSim data and real robot data on both RoboGSim and real robot platforms. The experimental results show that the RoboGSim data model can achieve zero-shot performance on the real robot, with results comparable to real robot data. Additionally, in experiments with novel perspectives and novel scenes, the RoboGSim data model performed even better on the real robot than the real robot data model. This not only helps reduce the sim2real gap but also addresses the limitations of real robot data collection, such as its single-source and high cost. We hope RoboGSim serves as a closed-loop simulator for fair comparison on policy learning. More information can be found on our project page https://robogsim.github.io/.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

31 of 44 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  9. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  10. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  11. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  12. 2023Gaussian Splatting SLAMcited by 410
19 more
  1. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  2. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 331
  3. 2016Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networkscited by 307
  4. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  5. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  6. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  7. 2024Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussianscited by 92
  8. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  9. 2018Reinforcement Learning with Perturbed Rewardscited by 76
  10. 2024GIM: Learning Generalizable Image Matcher From Internet Videoscited by 72
  11. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 63
  12. 2024Robo-GS: A Physics Consistent Spatial-Temporal Model for Robotic Arm with Hybrid Representationcited by 44
  13. 2023RoboHive: A Unified Framework for Robot Learningcited by 42
  14. 2022NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fieldscited by 41
  15. 2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 40
  16. 2023GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noisecited by 30
  17. 2024Differentiable Robot Renderingcited by 20
  18. 2020Policy Transfer via Kinematic Domain Randomization and Adaptationcited by 16
  19. 2023Domain Randomization for Sim2real Transfer of Automatically Generated Grasping Datasetscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 23.

50 papers.

Sort:

By area:

  1. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 27
  2. 2025ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesiscited by 22
  3. 2025PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulationcited by 22
  4. 2025VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotioncited by 20
  5. 2025Robot Learning from a Physical World Modelcited by 18
  6. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  7. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  8. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
  9. 2025Dexbotic: Open-Source Vision-Language-Action Toolboxcited by 12
  10. 2025Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robotscited by 11
  11. 2025OceanSim: A GPU-Accelerated Underwater Robot Perception Simulation Frameworkcited by 10
  12. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10
38 more
  1. 2026SimRecon: SimReady Compositional Scene Reconstruction from Real Videoscited by 9
  2. 2026Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulationcited by 9
  3. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 9
  4. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 8
  5. 2025Robot Learning from Any Imagescited by 7
  6. 2025TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulationcited by 7
  7. 2026TwinRL: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulationcited by 5
  8. 2025ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimationcited by 4
  9. 2025Bootstrapping Imitation Learning for Long-horizon Manipulation via Hierarchical Data Collection Spacecited by 3
  10. 2025ContraGS: Codebook-Condensed and Trainable Gaussian Splatting for Fast, Memory-Efficient Reconstructioncited by 3
  11. 2025Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulationcited by 3
  12. 2026TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkourcited by 3
  13. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 3
  14. 2025PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrationscited by 2
  15. 2025Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planningcited by 2
  16. 2025IGFuse: Interactive 3D Gaussian Scene Reconstruction via Multi-Scans Fusioncited by 2
  17. 2025Robust and High-Fidelity 3D Gaussian Splatting: Fusing Pose Priors and Geometry Constraints for Texture-Deficient Outdoor Scenescited by 2
  18. 2025AOMGen: Photoreal, Physics-Consistent Demonstration Generation for Articulated Object Manipulationcited by 2
  19. 2026ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientationcited by 2
  20. 2026VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenescited by 2
  21. 2026WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLNcited by 2
  22. 2025Interpretable Single-View 3D Gaussian Splatting using Unsupervised Hierarchical Disentangled Representation Learningcited by 1
  23. 2025DSG-World: Learning a 3D Gaussian World Model from Dual State Videoscited by 1
  24. 2025IGen: Scalable Data Generation for Robot Learning from Open-World Imagescited by 1
  25. 2025RecurGS: Interactive Scene Modeling via Discrete-State Recurrent Gaussian Fusioncited by 1
  26. 2026GEAR: GEometry-motion Alternating Refinement for Articulated Object Modeling with Gaussian Splattingcited by 1
  27. 2026From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluationcited by 1
  28. 2026Online Segment 3D Gaussians via Launching Virtual Dronescited by 1
  29. 2026R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Simcited by 1
  30. 2026Mirage2Matter: A Physically Grounded Gaussian World Model from Videocited by 0
  31. 2026Language-Guided Grasping under Partial Observation for Mobile Manipulation in Field Inspection and Maintenancecited by 0
  32. 2026Splat2Real: Novel-view Scaling for Physical AI with 3D Gaussian Splattingcited by 0
  33. 20263D Generation for Embodied AI and Robotic Simulation: A Surveycited by 0
  34. 2026A Principled Approach for Creating High-fidelity Synthetic Demonstrations for Imitation Learningcited by 0
  35. 2026ManiSplat: Manipulation Trajectory Synthesis from Monocular Video via Decoupled 3D Gaussian Splattingcited by 0
  36. 2026R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Modelscited by 0
  37. 2026SEES: A Self-Evolving Embodied System via Failure-Guided VLA Policy Adaptationcited by 0
  38. 2026Real2Gym: Building Gyms from Videos, Bringing Skills to Robotscited by 0