# DSG-World: Learning a 3D Gaussian World Model from Dual State Videos

arXiv: https://arxiv.org/abs/2506.05217 · HTML: https://mlsanity.com/paper/2506.05217/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 42 of its 44 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 30 of 42)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2311.14521/) (2023)
- [3D-VLA: A 3D Vision-Language-Action Generative World Model](https://mlsanity.com/paper/2403.09631/) (2024)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025)
- [Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding](https://mlsanity.com/paper/2311.18482/) (2023)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)
- [iVideoGPT: Interactive VideoGPTs are Scalable World Models](https://mlsanity.com/paper/2405.15223/) (2024)
- [OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding](https://mlsanity.com/paper/2406.02058/) (2024)
- [Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians](https://mlsanity.com/paper/2403.09434/) (2024)
- [SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting](https://mlsanity.com/paper/2409.10161/) (2024)
- [FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally](https://mlsanity.com/paper/2409.08270/) (2024)
- [InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior](https://mlsanity.com/paper/2404.11613/) (2024)
- [RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator](https://mlsanity.com/paper/2411.11839/) (2024)
- [Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation](https://mlsanity.com/paper/2504.13175/) (2025)
- [Robo-GS: A Physics Consistent Spatial-Temporal Model for Robotic Arm with Hybrid Representation](https://mlsanity.com/paper/2408.14873/) (2024)
- [VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation](https://mlsanity.com/paper/2411.09153/) (2024)
- [3D Gaussian Splatting in Robotics: A Survey](https://mlsanity.com/paper/2410.12262/) (2024)
- [Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination](https://mlsanity.com/paper/2412.14957/) (2024)
- [Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware](https://mlsanity.com/paper/2505.09601/) (2025)
- [Re3Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation](https://mlsanity.com/paper/2502.08645/) (2025)
- [Gaga: Group Any Gaussians via 3D-aware Memory Bank](https://mlsanity.com/paper/2404.07977/) (2024)
- [InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception](https://mlsanity.com/paper/2411.19235/) (2024)
- [Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics](https://mlsanity.com/paper/2410.08257/) (2024)

## Built on by (most cited first, 1 of 1)

- [DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors](https://mlsanity.com/paper/2607.00889/) (2026)

## Abstract (arXiv)

Building an efficient and physically consistent world model from limited observations is a long standing challenge in vision and robotics. Many existing world modeling pipelines are based on implicit generative models, which are hard to train and often lack 3D or physical consistency. On the other hand, explicit 3D methods built from a single state often require multi-stage processing-such as segmentation, background completion, and inpainting-due to occlusions. To address this, we leverage two perturbed observations of the same scene under different object configurations. These dual states offer complementary visibility, alleviating occlusion issues during state transitions and enabling more stable and complete reconstruction. In this paper, we present DSG-World, a novel end-to-end framework that explicitly constructs a 3D Gaussian World model from Dual State observations. Our approach builds dual segmentation-aware Gaussian fields and enforces bidirectional photometric and semantic consistency. We further introduce a pseudo intermediate state for symmetric alignment and design collaborative co-pruning trategies to refine geometric completeness. DSG-World enables efficient real-to-simulation transfer purely in the explicit Gaussian representation space, supporting high-fidelity rendering and object-level scene manipulation without relying on dense observations or multi-stage pipelines. Extensive experiments demonstrate strong generalization to novel views and scene states, highlighting the effectiveness of our approach for real-world 3D reconstruction and simulation.