mlsanityData to 7 Oct 2026

SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

arXiv 2409.10161 ↗First posted 16 Sep 2024cs.RO cs.AI cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

Sim2Real transfer, particularly for manipulation policies relying on RGB images, remains a critical challenge in robotics due to the significant domain shift between synthetic and real-world visual data. In this paper, we propose SplatSim, a novel framework that leverages Gaussian Splatting as the primary rendering primitive to reduce the Sim2Real gap for RGB-based manipulation policies. By replacing traditional mesh representations with Gaussian Splats in simulators, SplatSim produces highly photorealistic synthetic data while maintaining the scalability and cost-efficiency of simulation. We demonstrate the effectiveness of our framework by training manipulation policies within SplatSim and deploying them in the real world in a zero-shot manner, achieving an average success rate of 86.25%, compared to 97.5% for policies trained on real-world data. Videos can be found on our project page: https://splatsim.github.io

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 48 of its 58 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

48 of 58 references.

References · 48

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  5. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  7. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  8. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  9. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  10. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  11. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  12. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
36 more
  1. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  2. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulatorscited by 180
  3. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  4. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  5. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  6. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  7. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 131
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  9. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  10. 2023Extreme Parkour with Legged Robotscited by 120
  11. 2023Robot Parkour Learningcited by 115
  12. 2023ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robotscited by 105
  13. 2024Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splattingcited by 104
  14. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  15. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 98
  16. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  17. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  18. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  19. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  20. 2019Deep Drone Racing: From Simulation to Reality with Domain Randomizationcited by 63
  21. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  22. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  23. 2024Robo-GS: A Physics Consistent Spatial-Temporal Model for Robotic Arm with Hybrid Representationcited by 44
  24. 2023DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulationcited by 42
  25. 2022NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fieldscited by 41
  26. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  27. 2022Adapting Rapid Motor Adaptation for Bipedal Robotscited by 39
  28. 2024Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Roboticscited by 39
  29. 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 37
  30. 2024DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabricscited by 36
  31. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  32. 2023FlowBot++: Learning Generalized Articulated Objects Manipulation via Articulation Projectioncited by 34
  33. 2023Dynamic Handover: Throw and Catch with Bimanual Handscited by 33
  34. 2023HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulationcited by 27
  35. 2024Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networkscited by 12
  36. 2024Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 29; 2026: 30.

63 papers.

Sort:

By area:

  1. 2024RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulatorcited by 50
  2. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  3. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  4. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  5. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  6. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 29
  7. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 27
  8. 2024Neural Fields in Robotics: A Surveycited by 21
  9. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 19
  10. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  11. 2024SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuumcited by 13
  12. 2025Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixelscited by 12
38 more
  1. 2025Dexbotic: Open-Source Vision-Language-Action Toolboxcited by 12
  2. 2025Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robotscited by 11
  3. 2025EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Devicecited by 11
  4. 2025IGL-Nav: Incremental 3D Gaussian Localization for Image-goal Navigationcited by 10
  5. 2026PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generationcited by 9
  6. 2025WoMAP: World Models For Embodied Open-Vocabulary Object Localizationcited by 8
  7. 2025TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulationcited by 7
  8. 2025Is Single-View Mesh Reconstruction Ready for Robotics?cited by 6
  9. 2026ReaDy-Go: Real-to-Sim Dynamic 3D Gaussian Splatting Simulation for Environment-Specific Visual Navigation with Moving Obstaclescited by 5
  10. 2026Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observationscited by 5
  11. 2025SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learningcited by 4
  12. 2025Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interfacecited by 4
  13. 2025Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulationcited by 3
  14. 2026TTT-Parkour: Rapid Test-Time Training for Perceptive Robot Parkourcited by 3
  15. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 3
  16. 2025Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisionscited by 2
  17. 2025PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrationscited by 2
  18. 2025Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planningcited by 2
  19. 2025IGFuse: Interactive 3D Gaussian Scene Reconstruction via Multi-Scans Fusioncited by 2
  20. 2025RealMirror: A Comprehensive, Open-Source Vision-Language-Action Platform for Embodied AIcited by 2
  21. 2025AOMGen: Photoreal, Physics-Consistent Demonstration Generation for Articulated Object Manipulationcited by 2
  22. 2026ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientationcited by 2
  23. 2026VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenescited by 2
  24. 2025UAVTwin: Neural Digital Twins for UAVs using Gaussian Splattingcited by 1
  25. 2025DSG-World: Learning a 3D Gaussian World Model from Dual State Videoscited by 1
  26. 2025IGen: Scalable Data Generation for Robot Learning from Open-World Imagescited by 1
  27. 2026From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluationcited by 1
  28. 2026TriSplat: Simulation-Ready Feed-Forward 3D Scene Reconstructioncited by 1
  29. 2026GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environmentscited by 1
  30. 2026R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Simcited by 1
  31. 2026DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptationcited by 1
  32. 2024Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splattingcited by 0
  33. 2025FloPE: Flower Pose Estimation for Precision Pollinationcited by 0
  34. 2025ComplicitSplat: Downstream Models are Vulnerable to Blackbox Attacks by 3D Gaussian Splat Camouflagescited by 0
  35. 2026Splat2Real: Novel-view Scaling for Physical AI with 3D Gaussian Splattingcited by 0
  36. 20263D Generation for Embodied AI and Robotic Simulation: A Surveycited by 0
  37. 2026A Principled Approach for Creating High-fidelity Synthetic Demonstrations for Imitation Learningcited by 0
  38. 2026ManiSplat: Manipulation Trajectory Synthesis from Monocular Video via Decoupled 3D Gaussian Splattingcited by 0

50 most cited of 63 All 63 (JSON)