Learning Physics-Guided Residual Dynamics for Deformable Object Simulation
Not on the map. Why?
Abstract (arXiv)
Simulating deformable objects is essential for a wide range of robotic manipulation applications, yet accurately predicting their dynamics remains challenging. We propose Physics-Guided Residual Dynamics (PGRD), a hybrid simulation framework that combines the advantages of physics-based and learning-based approaches. Specifically, PGRD combines an optimizable spring-mass simulator as a backbone with a learned neural network that predicts residual corrections to the physics-based predictions. We adopt a velocity-based formulation to ensure stable simulation and a sliding-window transformer architecture to capture temporal dependencies. We show that PGRD produces more accurate results than both purely physics-based and learning-based methods on a set of diverse real-world deformable objects. We further demonstrate the utility of PGRD in two applications: manipulation planning via Model Predictive Control, including a language-conditioned setting with a generated goal image; and interactive simulation via action-conditioned video prediction by 3D Gaussian Splatting.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 65 of its 107 references that are in the map. First posted on arXiv 15 Jul 2026.
Builds on
65 of 107 references.
References · 65
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2024Depth Anything V2cited by 1,385
- 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
- 2020Learning Mesh-Based Simulation with Graph Networkscited by 840
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
- 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
- 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
- 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
- 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
38 more
- 2019Contrastive Learning of Structured World Modelscited by 206
- 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 205
- 2020Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solverscited by 201
- 2020Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Predictioncited by 137
- 2020Learning to Control PDEs with Differentiable Physicscited by 131
- 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 122
- 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
- 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
- 2024Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussianscited by 92
- 2018Flexible Neural Representation for Physics Predictioncited by 91
- 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 78
- 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
- 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 75
- 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
- 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
- 2018Propagation Networks for Model-Based Control Under Partial Observationcited by 64
- 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
- 2021DiffPD: Differentiable Projective Dynamicscited by 62
- 2021DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cuttingcited by 62
- 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
- 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
- 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 61
- 2023Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamicscited by 56
- 2024AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulationcited by 52
- 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
- 2019Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objectscited by 44
- 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
- 2022Differentiable Simulation of Soft Multi-body Systemscited by 39
- 2018Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncingcited by 36
- 2021NeuroBEM: Hybrid Aerodynamic Quadrotor Modelcited by 33
- 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
- 2022RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimationcited by 29
- 2021Conditional Neural Relational Inference for Interacting Systemscited by 28
- 2022Context is Everything: Implicit Identification for Dynamics Adaptationcited by 26
- 2025ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulationcited by 26
- 2025Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videoscited by 22
- 2021Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objectscited by 17
- 2022Offline-Online Learning of Deformation Model for Cable Manipulation with Graph Neural Networkscited by 17
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.