# Differentiable Physics-based System Identification for Robotic Manipulation of Elastoplastic Materials

arXiv: https://arxiv.org/abs/2411.00554 · HTML: https://mlsanity.com/paper/2411.00554/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [NeuralSim: Augmenting Differentiable Simulators with Neural Networks](https://mlsanity.com/paper/2011.04217/) (2020)
- [RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools](https://mlsanity.com/paper/2306.14447/) (2023)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation](https://mlsanity.com/paper/2303.02346/) (2023)
- [DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects](https://mlsanity.com/paper/2204.03139/) (2022)
- [ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation](https://mlsanity.com/paper/2203.06856/) (2022)
- [Rethinking Optimization with Differentiable Simulation from a Global Perspective](https://mlsanity.com/paper/2207.00167/) (2022)
- [Probabilistic Inference of Simulation Parameters via Parallel Differentiable Simulation](https://mlsanity.com/paper/2109.08815/) (2021)
- [Improving Physics-Augmented Continuum Neural Radiance Field-Based Geometry-Agnostic System Identification with Lagrangian Particle Optimization](https://mlsanity.com/paper/2406.04155/) (2024)
- [Manipulating Deformable Objects by Interleaving Prediction, Planning, and Control](https://mlsanity.com/paper/2001.09950/) (2020)
- [Model-free vision-based shaping of deformable plastic materials](https://mlsanity.com/paper/2001.11196/) (2020)
- [Vision-based system identification and 3D keypoint discovery using dynamics constraints](https://mlsanity.com/paper/2109.05928/) (2021)
- [Lattice-based shape tracking and servoing of elastic objects](https://mlsanity.com/paper/2209.01832/) (2022)
- [Real-time Model Predictive Control and System Identification Using Differentiable Physics Simulation](https://mlsanity.com/paper/2202.09834/) (2022)

## Built on by (most cited first, 6 of 6)

- [Learning Physics-Guided Residual Dynamics for Deformable Object Simulation](https://mlsanity.com/paper/2607.13451/) (2026)
- [Harnessing AI for Inverse Partial Differential Equation Problems: Past, Present, and Prospects](https://mlsanity.com/paper/2605.16966/) (2026)
- [PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics](https://mlsanity.com/paper/2607.20653/) (2026)
- [WorldString: Actionable World Representation](https://mlsanity.com/paper/2605.18743/) (2026)
- [Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators](https://mlsanity.com/paper/2605.20978/) (2026)
- [On the Numerical Reliability of Differentiable Physics-Based Optimization for Robotic Material Manipulation](https://mlsanity.com/paper/2609.34666/) (2026)

## Abstract (arXiv)

Robotic manipulation of volumetric elastoplastic deformable materials, from foods such as dough to construction materials like clay, is in its infancy, largely due to the difficulty of modelling and perception in a high-dimensional space. Simulating the dynamics of such materials is computationally expensive. It tends to suffer from inaccurately estimated physics parameters of the materials and the environment, impeding high-precision manipulation. Estimating such parameters from raw point clouds captured by optical cameras suffers further from heavy occlusions. To address this challenge, this work introduces a novel Differentiable Physics-based System Identification (DPSI) framework that enables a robot arm to infer the physics parameters of elastoplastic materials and the environment using simple manipulation motions and incomplete 3D point clouds, aligning the simulation with the real world. Extensive experiments show that with only a single real-world interaction, the estimated parameters, Young's modulus, Poisson's ratio, yield stress and friction coefficients, can accurately simulate visually and physically realistic deformation behaviours induced by unseen and long-horizon manipulation motions. Additionally, the DPSI framework inherently provides physically intuitive interpretations for the parameters in contrast to black-box approaches such as deep neural networks. The project is fully open-sourced via https://ianyangchina.github.io/SI4RP-data/.