mlsanityData to 7 Oct 2026

PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation

arXiv 2404.13026 ↗First posted 19 Apr 2024cs.CV cs.AIcited by 185

On the map: World models → Latent predictive world models

Abstract (arXiv)

Realistic object interactions are crucial for creating immersive virtual experiences, yet synthesizing realistic 3D object dynamics in response to novel interactions remains a significant challenge. Unlike unconditional or text-conditioned dynamics generation, action-conditioned dynamics requires perceiving the physical material properties of objects and grounding the 3D motion prediction on these properties, such as object stiffness. However, estimating physical material properties is an open problem due to the lack of material ground-truth data, as measuring these properties for real objects is highly difficult. We present PhysDreamer, a physics-based approach that endows static 3D objects with interactive dynamics by leveraging the object dynamics priors learned by video generation models. By distilling these priors, PhysDreamer enables the synthesis of realistic object responses to novel interactions, such as external forces or agent manipulations. We demonstrate our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study. PhysDreamer takes a step towards more engaging and realistic virtual experiences by enabling static 3D objects to dynamically respond to interactive stimuli in a physically plausible manner. See our project page at https://physdreamer.github.io/.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 56 of its 80 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

56 of 80 references.

References · 56

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  10. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  11. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  12. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
38 more
  1. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  2. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  3. 2020Convolutional Occupancy Networkscited by 760
  4. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  5. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  6. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  7. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  8. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  9. 2021Neural Fields in Visual Computing and Beyondcited by 539
  10. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  11. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  12. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  13. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  14. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 368
  15. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  16. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 339
  17. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  18. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  19. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  20. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  21. 2023Text-To-4D Dynamic Scene Generationcited by 204
  22. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 199
  23. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  24. 2020Robust Consistent Video Depth Estimationcited by 188
  25. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  26. 2023HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Samplingcited by 173
  27. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  28. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 145
  29. 2023DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splattingcited by 124
  30. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  31. 2023Generative Image Dynamicscited by 93
  32. 2023Structure-Aware Sparse-View X-ray 3D Reconstructioncited by 70
  33. 2023Forward Flow for Novel View Synthesis of Dynamic Scenescited by 67
  34. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  35. 2021Consistent Depth of Moving Objects in Videocited by 56
  36. 2023Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamicscited by 56
  37. 2023PIE-NeRF: Physics-based Interactive Elastodynamics with NeRFcited by 51
  38. 2022Differentiable Physics Simulation of Dynamics-Augmented Neural Objectscited by 45

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 75; 2026: 79.

185 papers.

Sort:

By area:

  1. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  4. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  5. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  6. 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 78
  7. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 77
  8. 2024Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusioncited by 76
  9. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  10. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  11. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  12. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
38 more
  1. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  2. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  3. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  4. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  5. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
  6. 2024DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priorscited by 40
  7. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  8. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  9. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  10. 2025RoboScape: Physics-informed Embodied World Modelcited by 38
  11. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  12. 2024PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulationcited by 36
  13. 2024Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstructioncited by 32
  14. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  15. 2024Neural Material Adaptor for Visual Grounding of Intrinsic Dynamicscited by 30
  16. 2024Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splattingcited by 30
  17. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  18. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  19. 2024Pathways on the Image Manifold: Image Editing via Video Generationcited by 27
  20. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 27
  21. 2024GASP: Gaussian Splatting for Physic-Based Simulationscited by 24
  22. 2025Generative Physical AI in Vision: A Surveycited by 24
  23. 2024Phys4DGen: Physics-Compliant 4D Generation with Multi-Material Composition Perceptioncited by 23
  24. 2025A Survey of Interactive Generative Videocited by 23
  25. 2025Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videoscited by 22
  26. 2025Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixelscited by 22
  27. 2024VividDream: Generating 3D Scene with Ambient Dynamicscited by 21
  28. 2025FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Videocited by 21
  29. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  30. 2025A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generationcited by 19
  31. 2025Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulationcited by 18
  32. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 18
  33. 20244Dynamic: Text-to-4D Generation with Hybrid Priorscited by 15
  34. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  35. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Modelscited by 15
  36. 2025PAT3D: Physics-Augmented Text-to-3D Scene Generationcited by 14
  37. 2025Articulated Kinematics Distillation from Video Diffusion Modelscited by 13
  38. 2025FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocitycited by 13

50 most cited of 185 All 185 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PhysReal: Learning Real-World Deformable Object Physics via Hybrid Constitutive Modeling
  2. 2026PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
  3. 2026PhysiFormer: Learning to Simulate Mechanics in World Space
  4. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suite
  5. 2026PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions
  6. 2025PhysX-3D: Physical-Grounded 3D Asset Generation
  7. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
  8. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI