# Predicting Physical Object Properties from Video

arXiv: https://arxiv.org/abs/2206.00930 · HTML: https://mlsanity.com/paper/2206.00930/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 2 Jun 2022.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [A Differentiable Physics Engine for Deep Learning in Robotics](https://mlsanity.com/paper/1611.01652/) (2016)
- [Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact](https://mlsanity.com/paper/2103.16021/) (2021)
- [Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor](https://mlsanity.com/paper/1704.03955/) (2017)
- [Cartman: The low-cost Cartesian Manipulator that won the Amazon Robotics Challenge](https://mlsanity.com/paper/1709.06283/) (2017)
- [Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing](https://mlsanity.com/paper/1810.02977/) (2018)
- [Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics](https://mlsanity.com/paper/2005.05659/) (2020)
- [Learning physical properties of liquid crystals with deep convolutional neural networks](https://mlsanity.com/paper/2004.01691/) (2020)
- [Real-time System Identification Using Deep Learning for Linear Processes with Application to Unmanned Aerial Vehicles](https://mlsanity.com/paper/2004.08603/) (2020)
- [Mass Estimation from Images using Deep Neural Network and Sparse Ground Truth](https://mlsanity.com/paper/1908.04387/) (2019)

## Built on by (most cited first, 2 of 2)

- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [Guiding Video Prediction with Explicit Procedural Knowledge](https://mlsanity.com/paper/2406.18220/) (2024)

## Abstract (arXiv)

We present a novel approach to estimating physical properties of objects from video. Our approach consists of a physics engine and a correction estimator. Starting from the initial observed state, object behavior is simulated forward in time. Based on the simulated and observed behavior, the correction estimator then determines refined physical parameters for each object. The method can be iterated for increased precision. Our approach is generic, as it allows for the use of an arbitrary - not necessarily differentiable - physics engine and correction estimator. For the latter, we evaluate both gradient-free hyperparameter optimization and a deep convolutional neural network. We demonstrate faster and more robust convergence of the learned method in several simulated 2D scenarios focusing on bin situations.