mlsanityData to 7 Oct 2026

gradSim: Differentiable simulation for system identification and visuomotor control

arXiv 2104.02646 ↗First posted 6 Apr 2021cs.CV cs.AI cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph -- spanning from the dynamics and through the rendering process -- enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 69 of its 123 references that are in the map. First posted on arXiv 6 Apr 2021.

Builds on

69 of 123 references.

References · 69

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  8. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  9. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  10. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  11. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  12. 2019Hamiltonian Neural Networkscited by 702
38 more
  1. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  2. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  3. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  4. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  5. 2019The frontier of simulation-based inferencecited by 490
  6. 2017Neural 3D Mesh Renderercited by 437
  7. 2021Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapescited by 426
  8. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  9. 2020Lagrangian Neural Networkscited by 388
  10. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  11. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  12. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 248
  13. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  14. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  15. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  16. 2019Symplectic ODE-Net: Learning Hamiltonian Dynamics with Controlcited by 183
  17. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  18. 2019Symplectic Recurrent Neural Networkscited by 148
  19. 2019Hamiltonian Generative Networkscited by 136
  20. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  21. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 107
  22. 2019Hamiltonian Graph Networks with ODE Integratorscited by 107
  23. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  24. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  25. 2018ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Roboticscited by 98
  26. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  27. 2017Visual Interaction Networkscited by 94
  28. 2020Learning Deformable Tetrahedral Meshes for 3D Reconstructioncited by 84
  29. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 78
  30. 2020Scalable Differentiable Physics for Learning and Controlcited by 72
  31. 2016Unsupervised Learning of 3D Structure from Imagescited by 67
  32. 2018SPNets: Differentiable Fluid Dynamics for Deep Neural Networkscited by 62
  33. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  34. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  35. 2020Visual Grounding of Learned Physical Modelscited by 54
  36. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  37. 2016SE3-Nets: Learning Rigid Body Motion using Deep Neural Networkscited by 48
  38. 2018RenderNet: A deep convolutional network for differentiable rendering from 3D shapescited by 42

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 13; 2023: 18; 2024: 17; 2025: 17; 2026: 29.

100 papers.

Sort:

By area:

  1. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 145
  2. 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 78
  3. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  4. 2023Adaptive Shells for Efficient Neural Radiance Field Renderingcited by 69
  5. 2021DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contactcited by 64
  6. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  7. 2023Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamicscited by 56
  8. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 53
  9. 2024GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulationcited by 48
  10. 2022DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Toolscited by 44
  11. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 38
  12. 2022State of the Art in Dense Monocular Non-Rigid 3D Reconstructioncited by 30
38 more
  1. 2024Neural Material Adaptor for Visual Grounding of Intrinsic Dynamicscited by 30
  2. 2022DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objectscited by 29
  3. 2022RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimationcited by 29
  4. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 29
  5. 2023Physics-Informed Machine Learning for Modeling and Control of Dynamical Systemscited by 28
  6. 2025Generative Physical AI in Vision: A Surveycited by 24
  7. 2023DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physicscited by 23
  8. 2022SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Renderingcited by 22
  9. 2025PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulationcited by 22
  10. 2025Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixelscited by 22
  11. 2022Rethinking Optimization with Differentiable Simulation from a Global Perspectivecited by 21
  12. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  13. 2025Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit with Diffusion Prior and Differentiable Physicscited by 19
  14. 2025Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulationcited by 18
  15. 2022Training Efficient Controllers via Analytic Policy Gradientcited by 16
  16. 2024Improving Physics-Augmented Continuum Neural Radiance Field-Based Geometry-Agnostic System Identification with Lagrangian Particle Optimizationcited by 16
  17. 2022Contact Points Discovery for Soft-Body Manipulations with Differentiable Physicscited by 14
  18. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
  19. 2023SoftMAC: Differentiable Soft Body Simulation with Forecast-based Contact Model and Two-way Coupling with Articulated Rigid Bodies and Clothescited by 9
  20. 2022TetGAN: A Convolutional Neural Network for Tetrahedral Mesh Generationcited by 8
  21. 2023Jade: A Differentiable Physics Engine for Articulated Rigid Bodies with Intersection-Free Frictional Contactcited by 7
  22. 2023Differentiable Visual Computing for Inverse Problems and Machine Learningcited by 7
  23. 2025MASIV: Toward Material-Agnostic System Identification from Videoscited by 7
  24. 2025Seeing the Wind from a Falling Leafcited by 7
  25. 2026ContactGaussian-WM: Learning Physics-Grounded World Model from Videoscited by 7
  26. 2023How Will It Drape Like? Capturing Fabric Mechanics from Depth Imagescited by 6
  27. 2024TorchDriveEnv: A Reinforcement Learning Benchmark for Autonomous Driving with Reactive, Realistic, and Diverse Non-Playable Characterscited by 6
  28. 2024Differentiable Physics-based System Identification for Robotic Manipulation of Elastoplastic Materialscited by 6
  29. 2026MOSIV: Multi-Object System Identification from Videoscited by 6
  30. 2021Vision-based system identification and 3D keypoint discovery using dynamics constraintscited by 5
  31. 2022Real2Sim2Real Transfer for Control of Cable-driven Robots via a Differentiable Physics Enginecited by 5
  32. 2023Neural Metamaterial Networks for Nonlinear Material Designcited by 5
  33. 2025Inferring Dynamic Physical Properties from Video Foundation Modelscited by 5
  34. 2022Differentiable Analog Quantum Computing for Optimization and Controlcited by 4
  35. 2023STAMP: Differentiable Task and Motion Planning via Stein Variational Gradient Descentcited by 4
  36. 2025ProJo4D: Progressive Joint Optimization for Sparse-View Inverse Physics Estimationcited by 4
  37. 2025Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challengescited by 4
  38. 2023GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policycited by 3

50 most cited of 100 All 100 (JSON)