mlsanityData to 14 Oct 2026

NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos

arXiv 2210.12352 ↗First posted 22 Oct 2022cs.CV cs.GR cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves as a reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with a differentiable physics simulator by designing a two-way conversion between the neural field and its corresponding hexahedral mesh, enabling us to estimate physics parameters from the source video by minimizing a cycle consistency loss. Our method also allows a user to interactively edit 3D objects from the source video by modifying the recovered hexahedral mesh, and propagating the operation back to the neural field representation. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches, and we provide extensive examples which demonstrate its ability to extract useful 3D representations from videos captured with consumer-grade cameras.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 54 of its 74 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

54 of 74 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  4. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  5. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  6. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  7. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  8. 2021Volume Rendering of Neural Implicit Surfacescited by 996
  9. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  10. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 861
  11. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 735
  12. 2020Implicit Geometric Regularization for Learning Shapescited by 710
38 more
  1. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  2. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
  3. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 646
  4. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  5. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 526
  6. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 471
  7. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 449
  8. 2017Neural 3D Mesh Renderercited by 437
  9. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  10. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  11. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  12. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  13. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  14. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
  15. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 217
  16. 2020Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solverscited by 205
  17. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 197
  18. 2019Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Imagescited by 164
  19. 2021Editable Free-viewpoint Video Using a Layered Neural Representationcited by 148
  20. 2017Learning Efficient Point Cloud Generation for Dense 3D Object Reconstructioncited by 133
  21. 2020Learning to Control PDEs with Differentiable Physicscited by 132
  22. 2022IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Imagescited by 111
  23. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 108
  24. 2021Neural Lumigraph Renderingcited by 105
  25. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  26. 2018ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Roboticscited by 98
  27. 2021NeuTex: Neural Texture Mapping for Volumetric Neural Renderingcited by 98
  28. 2021T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesiscited by 88
  29. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 82
  30. 2020Scalable Differentiable Physics for Learning and Controlcited by 73
  31. 2021DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contactcited by 65
  32. 2021DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cuttingcited by 64
  33. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 52
  34. 2021Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objectscited by 50
  35. 2020Neural Volume Rendering: NeRF And Beyondcited by 44
  36. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 39
  37. 2022Differentiable Simulation of Soft Multi-body Systemscited by 39
  38. 2022Virtual Elastic Objectscited by 35

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 13; 2024: 13; 2025: 14; 2026: 8.

50 papers.

Sort:

By area:

  1. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
  2. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 154
  3. 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 87
  4. 2022MonoNeRF: Learning a Generalizable Dynamic Radiance Field from Monocular Videoscited by 63
  5. 2023PIE-NeRF: Physics-based Interactive Elastodynamics with NeRFcited by 51
  6. 2024GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulationcited by 49
  7. 2023NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Cloudscited by 41
  8. 2023Control4D: Efficient 4D Portrait Editing with Textcited by 32
  9. 2022State of the Art in Dense Monocular Non-Rigid 3D Reconstructioncited by 30
  10. 2023NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videoscited by 28
  11. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  12. 2023BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fieldscited by 27
38 more
  1. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  2. 2025Generative Physical AI in Vision: A Surveycited by 24
  3. 2024GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMscited by 23
  4. 2023Dynamic Mesh-Aware Radiance Fieldscited by 21
  5. 2024Neural Fields in Robotics: A Surveycited by 21
  6. 2024Segment Any 4D Gaussianscited by 20
  7. 2025Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulationcited by 20
  8. 2023Learning 3D Particle-based Simulators from RGB-D Videoscited by 17
  9. 2024Deformable Radial Kernel Splattingcited by 17
  10. 2024ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editingcited by 12
  11. 2025Neural Radiance Fields for the Real World: A Surveycited by 12
  12. 2025TRACE: Learning 3D Gaussian Physical Dynamics from Multi-view Videoscited by 11
  13. 2025PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesiscited by 10
  14. 2025MASIV: Toward Material-Agnostic System Identification from Videoscited by 9
  15. 2023Dynamic Appearance Particle Neural Radiance Fieldcited by 6
  16. 2024Semantic Flow: Learning Semantic Field of Dynamic Scenes from Monocular Videoscited by 6
  17. 2026MOSIV: Multi-Object System Identification from Videoscited by 6
  18. 20234D-Editor: Interactive Object-level Editing in Dynamic Neural Radiance Fields via Semantic Distillationcited by 5
  19. 2024Scaling Face Interaction Graph Networks to Real World Scenescited by 5
  20. 2024FruitNinja: 3D Object Interior Texture Generation with Gaussian Splattingcited by 5
  21. 2025ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimationcited by 4
  22. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  23. 2025Affine-Equivariant Kernel Space Encoding for NeRF Editingcited by 4
  24. 2025Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentationcited by 4
  25. 2025Thin-Shell-SfT: Fine-Grained Monocular Non-rigid 3D Surface Tracking with Neural Deformation Fieldscited by 3
  26. 2026Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videoscited by 3
  27. 2026MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videoscited by 3
  28. 2026NeuROK: Generative 4D Neural Object Kinematicscited by 3
  29. 2025V2V3D: View-to-View Denoised 3D Reconstruction for Light-Field Microscopycited by 2
  30. 2026IRIS: Intersection-aware Ray-based Implicit Editable Scenescited by 2
  31. 2023FusedRF: Fusing Multiple Radiance Fieldscited by 1
  32. 2024PNR: Physics-informed Neural Representation for high-resolution LFM reconstructioncited by 1
  33. 2024NeRF-Texture: Synthesizing Neural Radiance Field Texturescited by 1
  34. 2025PSF-4D: A Progressive Sampling Framework for View Consistent 4D Editingcited by 1
  35. 2023UVA: Towards Unified Volumetric Avatar for View Synthesis, Pose rendering, Geometry and Texture Editingcited by 0
  36. 2026PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactionscited by 0
  37. 2026EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Videocited by 0
  38. 2026ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptionscited by 0