mlsanityData to 7 Oct 2026

Differentiable Physics Simulation of Dynamics-Augmented Neural Objects

arXiv 2210.09420 ↗First posted 17 Oct 2022cs.RO cs.AIcited by 45

Not on the map. Why?

Abstract (arXiv)

We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. This includes Neural Radiance Fields (NeRFs), and other related models. From the density field, we estimate the dynamical properties of the object, including its mass, center of mass, and inertia matrix. We then introduce a differentiable contact model based on the density field for computing normal and friction forces resulting from collisions. This allows a robot to autonomously build object models that are visually and dynamically accurate from still images and videos of objects in motion. The resulting Dynamics-Augmented Neural Objects (DANOs) are simulated with an existing differentiable simulation engine, Dojo, interacting with other standard simulation objects, such as spheres, planes, and robots specified as URDFs. A robot can use this simulation to optimize grasps and manipulation trajectories of neural objects, or to improve the neural object models through gradient-based real-to-simulation transfer. We demonstrate the pipeline to learn the coefficient of friction of a bar of soap from a real video of the soap sliding on a table. We also learn the coefficient of friction and mass of a Stanford bunny through interactions with a Panda robot arm from synthetic data, and we optimize trajectories in simulation for the Panda arm to push the bunny to a goal location.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

30 of 46 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  4. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  5. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  6. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  7. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  8. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  9. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 470
  10. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  11. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  12. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
18 more
  1. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  2. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  3. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  4. 2021HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fieldscited by 210
  5. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  6. 2021Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Controlcited by 106
  7. 2020Object-Centric Neural Scene Renderingcited by 90
  8. 2016Interactive Perception: Leveraging Action in Perception and Perception in Actioncited by 77
  9. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 77
  10. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  11. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  12. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  13. 2022Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Searchcited by 61
  14. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
  15. 2020ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representationscited by 38
  16. 2022DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objectscited by 29
  17. 2022ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulationcited by 24
  18. 2019A pressure field model for fast, robust approximation of net contact force and moment between nominally rigid objectscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 14; 2025: 11; 2026: 7.

45 papers.

Sort:

By area:

  1. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  2. 2024Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussianscited by 92
  3. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  4. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesiscited by 56
  5. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  6. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  7. 2024Neural Material Adaptor for Visual Grounding of Intrinsic Dynamicscited by 30
  8. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  9. 2023Dynamic Mesh-Aware Radiance Fieldscited by 21
  10. 2024Neural Fields in Robotics: A Surveycited by 21
  11. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 20
  12. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
33 more
  1. 2023Inferring Hybrid Neural Fluid Fields from Videoscited by 18
  2. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 18
  3. 2023Learning 3D Particle-based Simulators from RGB-D Videoscited by 17
  4. 2024Learning In-Hand Translation Using Tactile Skin With Shear and Normal Force Sensingcited by 13
  5. 2024EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splattingcited by 13
  6. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  7. 2024Learning rigid-body simulators over implicit shapes for large-scale scenes and visioncited by 12
  8. 2025Grounding Creativity in Physics: A Brief Survey of Physical Priors in AIGCcited by 11
  9. 2023Neural radiance fields in the industrial and robotics domain: applications, research opportunities and use casescited by 10
  10. 2023Multi-Object Manipulation via Object-Centric Neural Scattering Functionscited by 9
  11. 2024DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objectscited by 9
  12. 2023DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulationcited by 8
  13. 2024Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Videocited by 8
  14. 2024Scaling Face Interaction Graph Networks to Real World Scenescited by 5
  15. 2024Human Motion Prediction under Unexpected Perturbationcited by 5
  16. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  17. 2023GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policycited by 3
  18. 2023Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videoscited by 3
  19. 2023Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Renderingcited by 3
  20. 2025PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided LLMcited by 3
  21. 2025Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisionscited by 2
  22. 2025Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planningcited by 2
  23. 2023GenORM: Generalizable One-shot Rope Manipulation with Parameter-Aware Policycited by 1
  24. 2024DEF-oriCORN: efficient 3D scene understanding for robust language-directed manipulation without demonstrationscited by 1
  25. 2026Adaptive Volumetric Mechanical Property Fields Invariant to Resolutioncited by 1
  26. 2023Instance-Agnostic Geometry and Contact Dynamics Learningcited by 0
  27. 2024Deformable NeRF using Recursively Subdivided Tetrahedracited by 0
  28. 2025FoldPath: End-to-End Object-Centric Motion Generation via Modulated Implicit Pathscited by 0
  29. 2026MechVerse: Evaluating Physical Motion Consistency in Video Generation Modelscited by 0
  30. 2026PersistGS: Differentiable Physics for Object Permanence in 4D Gaussian Splattingcited by 0
  31. 2026SparseStreet: Sparse Gaussian Splatting for Real-Time Street Scene Simulationcited by 0
  32. 2026Differentiable Voxelization of Surface Representationscited by 0
  33. 2026ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecastingcited by 0