mlsanityData to 7 Oct 2026

Inferring Articulated Rigid Body Dynamics from RGBD Video

arXiv 2203.10488 ↗First posted 20 Mar 2022cs.RO cs.AI cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Being able to reproduce physical phenomena ranging from light interaction to contact mechanics, simulators are becoming increasingly useful in more and more application domains where real-world interaction or labeled data are difficult to obtain. Despite recent progress, significant human effort is needed to configure simulators to accurately reproduce real-world behavior. We introduce a pipeline that combines inverse rendering with differentiable simulation to create digital twins of real-world articulated mechanisms from depth or RGB videos. Our approach automatically discovers joint types and estimates their kinematic parameters, while the dynamic properties of the overall mechanism are tuned to attain physically accurate simulations. Control policies optimized in our derived simulation transfer successfully back to the original system, as we demonstrate on a simulated system. Further, our approach accurately reconstructs the kinematic tree of an articulated mechanism being manipulated by a robot, and highly nonlinear dynamics of a real-world coupled pendulum mechanism. Website: https://eric-heiden.github.io/video2sim

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 20 Mar 2022.

Builds on

23 of 40 references.

References · 23

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  3. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  4. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
  5. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  6. 2018Graph networks as learnable physics engines for inference and controlcited by 242
  7. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 205
  8. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  9. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  10. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  11. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 77
  12. 2020Scalable Differentiable Physics for Learning and Controlcited by 72
11 more
  1. 2020ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theorycited by 70
  2. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  3. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  4. 2021Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objectscited by 50
  5. 2014A Probabilistic Framework for Learning Kinematic Models of Articulated Objectscited by 40
  6. 2020Encoding Physical Constraints in Differentiable Newton-Euler Algorithmcited by 23
  7. 2020Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshopcited by 22
  8. 2021Distributional Depth-Based Estimation of Object Articulation Modelscited by 18
  9. 2020ADD: Analytically Differentiable Dynamics for Multi-Body Systems with Frictional Contactcited by 17
  10. 2021Probabilistic Inference of Simulation Parameters via Parallel Differentiable Simulationcited by 17
  11. 2019Physics-based Simulation of Continuous-Wave LIDAR for Localization, Calibration and Trackingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 2; 2026: 1.

10 papers.

  1. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 106
  2. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
  3. 2024DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videoscited by 63
  4. 2023Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Cloudscited by 34
  5. 2023Learning 3D Particle-based Simulators from RGB-D Videoscited by 17
  6. 2025Learning 3D-Gaussian Simulators from RGB Videoscited by 8
  7. 2023Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videoscited by 3
  8. 2024Learning Object Properties Using Robot Proprioception via Differentiable Robot-Object Interactioncited by 3
  9. 2025Generative 4D Scene Gaussian Splatting with Object View-Synthesis Priorscited by 1
  10. 2026URDF Synthesis from RGB-D Sequences via Differentiable Joint Inference and Energy-Consistent Verificationcited by 0