# PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification

arXiv: https://arxiv.org/abs/2303.05512 · HTML: https://mlsanity.com/paper/2303.05512/

## Key facts

- Cited by 154 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting](https://mlsanity.com/paper/2105.12244/) (2021)
- [DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions](https://mlsanity.com/paper/1906.03853/) (2019)
- [Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data](https://mlsanity.com/paper/2206.06577/) (2022)
- [NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos](https://mlsanity.com/paper/2210.12352/) (2022)
- [Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video](https://mlsanity.com/paper/1905.11169/) (2019)
- [Differentiable Simulation of Soft Multi-body Systems](https://mlsanity.com/paper/2205.01758/) (2022)
- [NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields](https://mlsanity.com/paper/2203.01762/) (2022)
- [Virtual Elastic Objects](https://mlsanity.com/paper/2201.04623/) (2022)
- [RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation](https://mlsanity.com/paper/2205.05678/) (2022)
- [Differentiable Implicit Soft-Body Physics](https://mlsanity.com/paper/2102.05791/) (2021)
- [Vision-based system identification and 3D keypoint discovery using dynamics constraints](https://mlsanity.com/paper/2109.05928/) (2021)
- [Complex Locomotion Skill Learning via Differentiable Physics](https://mlsanity.com/paper/2206.02341/) (2022)

## Built on by (most cited first, 30 of 154)

- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality](https://mlsanity.com/paper/2401.16663/) (2024)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [Flexible Isosurface Extraction for Gradient-Based Mesh Optimization](https://mlsanity.com/paper/2308.05371/) (2023)
- [3D Gaussian as a New Era: A Survey](https://mlsanity.com/paper/2402.07181/) (2024)
- [Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians](https://mlsanity.com/paper/2403.09434/) (2024)
- [PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos](https://mlsanity.com/paper/2503.17973/) (2025)
- [PhysGen3D: Crafting a Miniature Interactive World from a Single Image](https://mlsanity.com/paper/2503.20746/) (2025)
- [PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF](https://mlsanity.com/paper/2311.13099/) (2023)
- [GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation](https://mlsanity.com/paper/2406.14927/) (2024)
- [Physical Property Understanding from Language-Embedded Feature Fields](https://mlsanity.com/paper/2404.04242/) (2024)
- [PhyRecon: Physically Plausible Neural Scene Reconstruction](https://mlsanity.com/paper/2404.16666/) (2024)
- [PhysMotion: Physics-Grounded Dynamics From a Single Image](https://mlsanity.com/paper/2411.17189/) (2024)
- [Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination](https://mlsanity.com/paper/2412.14957/) (2024)
- [WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions](https://mlsanity.com/paper/2505.18151/) (2025)
- [Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling](https://mlsanity.com/paper/2410.18912/) (2024)
- [Physically Compatible 3D Object Modeling from a Single Image](https://mlsanity.com/paper/2405.20510/) (2024)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Gaussian Splashing: Unified Particles for Versatile Motion Synthesis and Rendering](https://mlsanity.com/paper/2401.15318/) (2024)
- [PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation](https://mlsanity.com/paper/2411.14423/) (2024)
- [Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics](https://mlsanity.com/paper/2410.08257/) (2024)
- [Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting](https://mlsanity.com/paper/2411.12789/) (2024)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions](https://mlsanity.com/paper/2511.04665/) (2025)
- [Physics-Informed Computer Vision: A Review and Perspectives](https://mlsanity.com/paper/2305.18035/) (2023)
- [NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos](https://mlsanity.com/paper/2312.06398/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [AutoVFX: Physically Realistic Video Editing from Natural Language Instructions](https://mlsanity.com/paper/2411.02394/) (2024)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)

## Abstract (arXiv)

Existing approaches to system identification (estimating the physical parameters of an object) from videos assume known object geometries. This precludes their applicability in a vast majority of scenes where object geometries are complex or unknown. In this work, we aim to identify parameters characterizing a physical system from a set of multi-view videos without any assumption on object geometry or topology. To this end, we propose "Physics Augmented Continuum Neural Radiance Fields" (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. We design PAC-NeRF to only ever produce physically plausible states by enforcing the neural radiance field to follow the conservation laws of continuum mechanics. For this, we design a hybrid Eulerian-Lagrangian representation of the neural radiance field, i.e., we use the Eulerian grid representation for NeRF density and color fields, while advecting the neural radiance fields via Lagrangian particles. This hybrid Eulerian-Lagrangian representation seamlessly blends efficient neural rendering with the material point method (MPM) for robust differentiable physics simulation. We validate the effectiveness of our proposed framework on geometry and physical parameter estimation over a vast range of materials, including elastic bodies, plasticine, sand, Newtonian and non-Newtonian fluids, and demonstrate significant performance gain on most tasks.