mlsanityData to 7 Oct 2026

Vision-based system identification and 3D keypoint discovery using dynamics constraints

arXiv 2109.05928 ↗First posted 13 Sep 2021cs.CV cs.AI stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper introduces V-SysId, a novel method that enables simultaneous keypoint discovery, 3D system identification, and extrinsic camera calibration from an unlabeled video taken from a static camera, using only the family of equations of motion of the object of interest as weak supervision. V-SysId takes keypoint trajectory proposals and alternates between maximum likelihood parameter estimation and extrinsic camera calibration, before applying a suitable selection criterion to identify the track of interest. This is then used to train a keypoint tracking model using supervised learning. Results on a range of settings (robotics, physics, physiology) highlight the utility of this approach.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

32 of 59 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
  4. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  5. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  6. 2019Hamiltonian Neural Networkscited by 702
  7. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 335
  8. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  9. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  10. 2018LF-Net: Learning Local Features from Imagescited by 151
  11. 2019Hamiltonian Generative Networkscited by 136
  12. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 113
20 more
  1. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  2. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  3. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  4. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  5. 2018Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoningcited by 79
  6. 2020Scalable Differentiable Physics for Learning and Controlcited by 72
  7. 2019DeepSFM: Structure From Motion Via Deep Bundle Adjustmentcited by 70
  8. 2019Learning Compositional Koopman Operators for Model-Based Controlcited by 67
  9. 2020Visual Grounding of Learned Physical Modelscited by 54
  10. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  11. 2019Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Videocited by 43
  12. 2020Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Controlcited by 38
  13. 2020Deep Keypoint-Based Camera Pose Estimation with Geometric Constraintscited by 29
  14. 2020NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spacescited by 15
  15. 2017Model-Based Control Using Koopman Operatorscited by 14
  16. 2020Unsupervised Object Keypoint Learning using Local Spatial Predictabilitycited by 13
  17. 2020Learning to Identify Physical Parameters from Video Using Differentiable Physicscited by 11
  18. 2020Adversarial Generation of Informative Trajectories for Dynamics System Identificationcited by 3
  19. 2019Self-supervised Learning of Interpretable Keypoints from Unlabelled Videoscited by 2
  20. 2017CoMaL Tracking: Tracking Points at the Object Boundariescited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 1; 2025: 1.

5 papers.

  1. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 145
  2. 2022Neural Implicit Representations for Physical Parameter Inference from a Single Videocited by 19
  3. 2024Differentiable Physics-based System Identification for Robotic Manipulation of Elastoplastic Materialscited by 6
  4. 2022KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Imagescited by 0
  5. 2025Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Controlcited by 0