# Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data

arXiv: https://arxiv.org/abs/2506.04120 · HTML: https://mlsanity.com/paper/2506.04120/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2025.

## Built on by (most cited first, 5 of 5)

- [ContactGaussian-WM: Learning Physics-Grounded World Model from Videos](https://mlsanity.com/paper/2602.11021/) (2026)
- [Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulation](https://mlsanity.com/paper/2510.11689/) (2025)
- [Real2Gym: Building Gyms from Videos, Bringing Skills to Robots](https://mlsanity.com/paper/2609.37089/) (2026)
- [Real-Time Physics Simulation with Dynamic Mesh-Gaussian Reconstructions](https://mlsanity.com/paper/2606.00444/) (2026)
- [PersistGS: Differentiable Physics for Object Permanence in 4D Gaussian Splatting](https://mlsanity.com/paper/2606.03479/) (2026)

## Abstract (arXiv)

Creating accurate, physical simulations directly from real-world robot motion holds great value for safe, scalable, and affordable robot learning, yet remains exceptionally challenging. Real robot data suffers from occlusions, noisy camera poses, dynamic scene elements, which hinder the creation of geometrically accurate and photorealistic digital twins of unseen objects. We introduce a novel real-to-sim framework tackling all these challenges at once. Our key insight is a hybrid scene representation merging the photorealistic rendering of 3D Gaussian Splatting with explicit object meshes suitable for physics simulation within a single representation. We propose an end-to-end optimization pipeline that leverages differentiable rendering and differentiable physics within MuJoCo to jointly refine all scene components - from object geometry and appearance to robot poses and physical parameters - directly from raw and imprecise robot trajectories. This unified optimization allows us to simultaneously achieve high-fidelity object mesh reconstruction, generate photorealistic novel views, and perform annotation-free robot pose calibration. We demonstrate the effectiveness of our approach both in simulation and on challenging real-world sequences using an ALOHA 2 bi-manual manipulator, enabling more practical and robust real-to-simulation pipelines.