# Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics

arXiv: https://arxiv.org/abs/2406.10788 · HTML: https://mlsanity.com/paper/2406.10788/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 0 of its 19 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Built on by (most cited first, 30 of 40)

- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation](https://mlsanity.com/paper/2601.03782/) (2026)
- [SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting](https://mlsanity.com/paper/2409.10161/) (2024)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination](https://mlsanity.com/paper/2412.14957/) (2024)
- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos](https://mlsanity.com/paper/2506.15680/) (2025)
- [Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels](https://mlsanity.com/paper/2508.17437/) (2025)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [Robot Learning from a Physical World Model](https://mlsanity.com/paper/2511.07416/) (2025)
- [Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation](https://mlsanity.com/paper/2603.16669/) (2026)
- [GSCodec Studio: A Modular Framework for Gaussian Splat Compression](https://mlsanity.com/paper/2506.01822/) (2025)
- [PINGS: Gaussian Splatting Meets Distance Fields within a Point-Based Implicit Neural Map](https://mlsanity.com/paper/2502.05752/) (2025)
- [Learning 3D-Gaussian Simulators from RGB Videos](https://mlsanity.com/paper/2503.24009/) (2025)
- [TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulation](https://mlsanity.com/paper/2512.19390/) (2025)
- [Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations](https://mlsanity.com/paper/2604.27106/) (2026)
- [Is Single-View Mesh Reconstruction Ready for Robotics?](https://mlsanity.com/paper/2505.17966/) (2025)
- [Scaling Laws in Scientific Discovery with AI and Robot Scientists](https://mlsanity.com/paper/2503.22444/) (2025)
- [SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLM](https://mlsanity.com/paper/2603.23386/) (2026)
- [ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimation](https://mlsanity.com/paper/2506.05317/) (2025)
- [ContraGS: Codebook-Condensed and Trainable Gaussian Splatting for Fast, Memory-Efficient Reconstruction](https://mlsanity.com/paper/2509.03775/) (2025)
- [Towards a Science Exocortex](https://mlsanity.com/paper/2406.17809/) (2024)
- [Motion Blender Gaussian Splatting for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2503.09040/) (2025)
- [Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation](https://mlsanity.com/paper/2510.07975/) (2025)
- [D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping](https://mlsanity.com/paper/2603.01151/) (2026)
- [Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisions](https://mlsanity.com/paper/2503.11269/) (2025)
- [Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning](https://mlsanity.com/paper/2505.14938/) (2025)
- [Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)](https://mlsanity.com/paper/2509.06191/) (2025)
- [FreeForm: Reduced-Order Deformable Simulation from Particle-Based Skinning Eigenmodes](https://mlsanity.com/paper/2605.29318/) (2026)
- [Physically Viable World Models: A Case for Query-Conditioned Embodied AI](https://mlsanity.com/paper/2605.30542/) (2026)

## Abstract (arXiv)

For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation - modelling geometry, physics, and visual observations - that informs perception, planning, and control algorithms. We propose a novel dual Gaussian-Particle representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates visual forces that correct the particle positions while respecting known physical constraints. By integrating predictive physical modelling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. Our system runs in realtime at 30Hz using only 3 cameras. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/.