# WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions

arXiv: https://arxiv.org/abs/2505.18151 · HTML: https://mlsanity.com/paper/2505.18151/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 75 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 63)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [Text-To-4D Dynamic Scene Generation](https://mlsanity.com/paper/2301.11280/) (2023)
- [WonderWorld: Interactive 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2406.09394/) (2024)
- [DreamGaussian4D: Generative 4D Gaussian Splatting](https://mlsanity.com/paper/2312.17142/) (2023)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling](https://mlsanity.com/paper/2311.17984/) (2023)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)

## Built on by (most cited first, 30 of 45)

- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation](https://mlsanity.com/paper/2601.03782/) (2026)
- [Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals](https://mlsanity.com/paper/2505.19386/) (2025)
- [PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation](https://mlsanity.com/paper/2509.20358/) (2025)
- [NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics](https://mlsanity.com/paper/2509.21309/) (2025)
- [Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels](https://mlsanity.com/paper/2508.17437/) (2025)
- [RealWonder: Real-Time Physical Action-Conditioned Video Generation](https://mlsanity.com/paper/2603.05449/) (2026)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories](https://mlsanity.com/paper/2602.14941/) (2026)
- [Physical Simulator In-the-Loop Video Generation](https://mlsanity.com/paper/2603.06408/) (2026)
- [Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models](https://mlsanity.com/paper/2603.13215/) (2026)
- [PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding](https://mlsanity.com/paper/2511.20562/) (2025)
- [Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goals](https://mlsanity.com/paper/2601.05848/) (2026)
- [PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generation](https://mlsanity.com/paper/2602.04876/) (2026)
- [MemLearner: Learning to Query Context memory for Video World Models](https://mlsanity.com/paper/2606.31734/) (2026)
- [SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation](https://mlsanity.com/paper/2512.03350/) (2025)
- [Choreographing a World of Dynamic Objects](https://mlsanity.com/paper/2601.04194/) (2026)
- [SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agents](https://mlsanity.com/paper/2603.08403/) (2026)
- [PhyCo: Learning Controllable Physical Priors for Generative Motion](https://mlsanity.com/paper/2604.28169/) (2026)
- [GS-Agent: Creating 4D Physical Worlds With Generative Simulation](https://mlsanity.com/paper/2607.21522/) (2026)
- [SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation](https://mlsanity.com/paper/2602.02402/) (2026)
- [PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment](https://mlsanity.com/paper/2603.13770/) (2026)
- [PhysOmni: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction](https://mlsanity.com/paper/2605.20290/) (2026)
- [NeuROK: Generative 4D Neural Object Kinematics](https://mlsanity.com/paper/2605.30347/) (2026)
- [Learning Explicit Physical Parameter Control and Benchmarking for Video Generation](https://mlsanity.com/paper/2607.18924/) (2026)
- [Learning to Generate Rigid Body Interactions with Video Diffusion Models](https://mlsanity.com/paper/2510.02284/) (2025)
- [MoRight: Motion Control Done Right](https://mlsanity.com/paper/2604.07348/) (2026)
- [Envisioning the Future, One Step at a Time](https://mlsanity.com/paper/2604.09527/) (2026)
- [DeformMaster: An Interactive Physics-Neural World Model for Deformable Objects from Videos](https://mlsanity.com/paper/2605.09586/) (2026)
- [Inferring Compositional 4D Scenes without Ever Seeing One](https://mlsanity.com/paper/2512.05272/) (2025)

## Abstract (arXiv)

WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elastic dynamics, WonderPlay features a hybrid generative simulator to synthesize a wide range of 3D dynamics. The hybrid generative simulator first uses a physics solver to simulate coarse 3D dynamics, which subsequently conditions a video generator to produce a video with finer, more realistic motion. The generated video is then used to update the simulated dynamic 3D scene, closing the loop between the physics solver and the video generator. This approach enables intuitive user control to be combined with the accurate dynamics of physics-based simulators and the expressivity of diffusion-based video generators. Experimental results demonstrate that WonderPlay enables users to interact with various scenes of diverse content, including cloth, sand, snow, liquid, smoke, elastic, and rigid bodies -- all using a single image input. Code will be made public. Project website: https://kyleleey.github.io/WonderPlay/