# Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion

arXiv: https://arxiv.org/abs/2406.04338 · HTML: https://mlsanity.com/paper/2406.04338/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 21 of its 69 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 21 of 21)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model](https://mlsanity.com/paper/2403.05034/) (2024)
- [DreamGaussian4D: Generative 4D Gaussian Splatting](https://mlsanity.com/paper/2312.17142/) (2023)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models](https://mlsanity.com/paper/2312.13763/) (2023)
- [DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting](https://mlsanity.com/paper/2312.00112/) (2023)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior](https://mlsanity.com/paper/2312.06655/) (2023)
- [Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generation](https://mlsanity.com/paper/2403.09625/) (2024)

## Built on by (most cited first, 30 of 80)

- [ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model](https://mlsanity.com/paper/2408.16767/) (2024)
- [DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion](https://mlsanity.com/paper/2411.04928/) (2024)
- [HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels](https://mlsanity.com/paper/2507.21809/) (2025)
- [Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals](https://mlsanity.com/paper/2505.19386/) (2025)
- [WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions](https://mlsanity.com/paper/2505.18151/) (2025)
- [Video-T1: Test-Time Scaling for Video Generation](https://mlsanity.com/paper/2503.18942/) (2025)
- [RoboScape: Physics-informed Embodied World Model](https://mlsanity.com/paper/2506.23135/) (2025)
- [PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation](https://mlsanity.com/paper/2411.14423/) (2024)
- [Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting](https://mlsanity.com/paper/2411.12789/) (2024)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step](https://mlsanity.com/paper/2504.01956/) (2025)
- [GASP: Gaussian Splatting for Physic-Based Simulations](https://mlsanity.com/paper/2409.05819/) (2024)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [Phys4DGen: Physics-Compliant 4D Generation with Multi-Material Composition Perception](https://mlsanity.com/paper/2411.16800/) (2024)
- [RealWonder: Real-Time Physical Action-Conditioned Video Generation](https://mlsanity.com/paper/2603.05449/) (2026)
- [Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation](https://mlsanity.com/paper/2506.06440/) (2025)
- [PhyCAGE: Physically Plausible Compositional 3D Asset Generation from a Single Image](https://mlsanity.com/paper/2411.18548/) (2024)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [DecoupledGaussian: Object-Scene Decoupling for Physics-Based Interaction](https://mlsanity.com/paper/2503.05484/) (2025)
- [Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning](https://mlsanity.com/paper/2608.27549/) (2026)
- [FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity](https://mlsanity.com/paper/2506.07865/) (2025)
- [Grounding Creativity in Physics: A Brief Survey of Physical Priors in AIGC](https://mlsanity.com/paper/2502.07007/) (2025)
- [TRACE: Learning 3D Gaussian Physical Dynamics from Multi-view Videos](https://mlsanity.com/paper/2508.09811/) (2025)
- [PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesis](https://mlsanity.com/paper/2508.13911/) (2025)
- [MASIV: Toward Material-Agnostic System Identification from Videos](https://mlsanity.com/paper/2508.01112/) (2025)
- [Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goals](https://mlsanity.com/paper/2601.05848/) (2026)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [PhysRig: Differentiable Physics-Based Skinning and Rigging Framework for Realistic Articulated Object Modeling](https://mlsanity.com/paper/2506.20936/) (2025)
- [CFG-Ctrl: Control-Based Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2603.03281/) (2026)
- [Seeing the Wind from a Falling Leaf](https://mlsanity.com/paper/2512.00762/) (2025)

## Abstract (arXiv)

In recent years, there has been rapid development in 3D generation models, opening up new possibilities for applications such as simulating the dynamic movements of 3D objects and customizing their behaviors. However, current 3D generative models tend to focus only on surface features such as color and shape, neglecting the inherent physical properties that govern the behavior of objects in the real world. To accurately simulate physics-aligned dynamics, it is essential to predict the physical properties of materials and incorporate them into the behavior prediction process. Nonetheless, predicting the diverse materials of real-world objects is still challenging due to the complex nature of their physical attributes. In this paper, we propose Physics3D, a novel method for learning various physical properties of 3D objects through a video diffusion model. Our approach involves designing a highly generalizable physical simulation system based on a viscoelastic material model, which enables us to simulate a wide range of materials with high-fidelity capabilities. Moreover, we distill the physical priors from a video diffusion model that contains more understanding of realistic object materials. Extensive experiments demonstrate the effectiveness of our method with both elastic and plastic materials. Physics3D shows great potential for bridging the gap between the physical world and virtual neural space, providing a better integration and application of realistic physical principles in virtual environments. Project page: https://liuff19.github.io/Physics3D.