# High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting

arXiv: https://arxiv.org/abs/2510.10637 · HTML: https://mlsanity.com/paper/2510.10637/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2025.

## Built on by (most cited first, 14 of 14)

- [Act2Goal: From World Model To General Goal-conditioned Policy](https://mlsanity.com/paper/2512.23541/) (2025)
- [DragMesh: Interactive 3D Generation Made Easy](https://mlsanity.com/paper/2512.06424/) (2025)
- [Articulat3D: Reconstructing Articulated Digital Twins From Monocular Videos with Geometric and Motion Constraints](https://mlsanity.com/paper/2603.11606/) (2026)
- [Physics-Aware 3D Gaussian Editing for Driving Scene Generation](https://mlsanity.com/paper/2605.25373/) (2026)
- [DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation](https://mlsanity.com/paper/2608.29078/) (2026)
- [Moving Light Adaptive Colonoscopy Reconstruction via Illumination-Attenuation-Aware 3D Gaussian Splatting](https://mlsanity.com/paper/2510.18739/) (2025)
- [Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment](https://mlsanity.com/paper/2603.28475/) (2026)
- [CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment](https://mlsanity.com/paper/2604.05484/) (2026)
- [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026)
- [KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos](https://mlsanity.com/paper/2609.21229/) (2026)
- [RoboCousin: Build Your Own Simulation Playground for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2609.08339/) (2026)
- [ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluation](https://mlsanity.com/paper/2609.23910/) (2026)
- [F4R: Failure-Driven Recognition, Reconstruction, Refinement, and Redeployment for Continual Robot Self-Improvement](https://mlsanity.com/paper/2609.35575/) (2026)
- [Splat2Act: From Captured Scenes to Personalized VLA Manipulation](https://mlsanity.com/paper/2610.13945/) (2026)

## Abstract (arXiv)

The scalability of robotic learning is fundamentally bottlenecked by the significant cost and labor of real-world data collection. While simulated data offers a scalable alternative, it often fails to generalize to the real world due to significant gaps in visual appearance, physical properties, and object interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real framework that converts multi-view real-world images into scalable, high-fidelity, and physically interactive simulation environments for robotic manipulation. Our approach reconstructs scenes using a hybrid representation: 3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the environment, while mesh primitives for interactive objects ensure accurate physics simulation. Crucially, we pioneer the use of a Multi-modal Large Language Model (MLLM) to automate the creation of physically plausible, articulated assets. The MLLM analyzes visual data to infer not only physical properties (e.g., density, stiffness) but also complex kinematic structures (e.g., hinges, sliding rails) of objects. We demonstrate that policies trained entirely on data generated by RoboSimGS achieve successful zero-shot sim-to-real transfer across a diverse set of real-world manipulation tasks. Furthermore, data from RoboSimGS significantly enhances the performance and generalization capabilities of SOTA methods. Our results validate RoboSimGS as a powerful and scalable solution for bridging the sim-to-real gap.