# Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects

arXiv: https://arxiv.org/abs/1911.06283 · HTML: https://mlsanity.com/paper/1911.06283/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2019.

## Built on by (most cited first, 30 of 45)

- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos](https://mlsanity.com/paper/2506.15680/) (2025)
- [ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation](https://mlsanity.com/paper/2203.06856/) (2022)
- [DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics](https://mlsanity.com/paper/2304.03223/) (2023)
- [Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics](https://mlsanity.com/paper/2205.02835/) (2022)
- [Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies](https://mlsanity.com/paper/2107.08942/) (2021)
- [Physics perception in sloshing scenes with guaranteed thermodynamic consistency](https://mlsanity.com/paper/2106.13301/) (2021)
- [Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning](https://mlsanity.com/paper/2204.00117/) (2022)
- [Deep Reinforcement Learning Based on Local GNN for Goal-conditioned Deformable Object Rearranging](https://mlsanity.com/paper/2302.10446/) (2023)
- [SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation](https://mlsanity.com/paper/2309.08728/) (2023)
- [Deformable Linear Object Prediction Using Locally Linear Latent Dynamics](https://mlsanity.com/paper/2103.14184/) (2021)
- [Disentangling Dense Multi-Cable Knots](https://mlsanity.com/paper/2106.02252/) (2021)
- [DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics](https://mlsanity.com/paper/2312.06408/) (2023)
- [Deformable One-Dimensional Object Detection for Routing and Manipulation](https://mlsanity.com/paper/2201.06775/) (2022)
- [TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach](https://mlsanity.com/paper/2407.03245/) (2024)
- [MaxMI: A Maximal Mutual Information Criterion for Manipulation Concept Discovery](https://mlsanity.com/paper/2407.15086/) (2024)
- [DexDLO: Learning Goal-Conditioned Dexterous Policy for Dynamic Manipulation of Deformable Linear Objects](https://mlsanity.com/paper/2312.15204/) (2023)
- [GenDOM: Generalizable One-shot Deformable Object Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2309.09051/) (2023)
- [A Systematic Literature Review of Computer Vision Applications in Robotized Wire Harness Assembly](https://mlsanity.com/paper/2309.13744/) (2023)
- [InfoCon: Concept Discovery with Generative and Discriminative Informativeness](https://mlsanity.com/paper/2404.10606/) (2024)
- [Keep it Simple: Data-efficient Learning for Controlling Complex Systems with Simple Models](https://mlsanity.com/paper/2102.02493/) (2021)
- [All You Need is LUV: Unsupervised Collection of Labeled Images using Invisible UV Fluorescent Indicators](https://mlsanity.com/paper/2203.04566/) (2022)
- [Detection and Physical Interaction with Deformable Linear Objects](https://mlsanity.com/paper/2205.08041/) (2022)
- [ISCUTE: Instance Segmentation of Cables Using Text Embedding](https://mlsanity.com/paper/2402.11996/) (2024)
- [Offline Goal-Conditioned Reinforcement Learning for Shape Control of Deformable Linear Objects](https://mlsanity.com/paper/2403.10290/) (2024)
- [Learning Physics-Guided Residual Dynamics for Deformable Object Simulation](https://mlsanity.com/paper/2607.13451/) (2026)
- [Bimanual rope manipulation skill synthesis through context dependent correction policy learning from human demonstration](https://mlsanity.com/paper/2209.13850/) (2022)
- [GenORM: Generalizable One-shot Rope Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2306.09872/) (2023)

## Abstract (arXiv)

We demonstrate model-based, visual robot manipulation of linear deformable objects. Our approach is based on a state-space representation of the physical system that the robot aims to control. This choice has multiple advantages, including the ease of incorporating physics priors in the dynamics model and perception model, and the ease of planning manipulation actions. In addition, physical states can naturally represent object instances of different appearances. Therefore, dynamics in the state space can be learned in one setting and directly used in other visually different settings. This is in contrast to dynamics learned in pixel space or latent space, where generalization to visual differences are not guaranteed. Challenges in taking the state-space approach are the estimation of the high-dimensional state of a deformable object from raw images, where annotations are very expensive on real data, and finding a dynamics model that is both accurate, generalizable, and efficient to compute. We are the first to demonstrate self-supervised training of rope state estimation on real images, without requiring expensive annotations. This is achieved by our novel self-supervising learning objective, which is generalizable across a wide range of visual appearances. With estimated rope states, we train a fast and differentiable neural network dynamics model that encodes the physics of mass-spring systems. Our method has a higher accuracy in predicting future states compared to models that do not involve explicit state estimation and do not use any physics prior, while only using 3% of training data. We also show that our approach achieves more efficient manipulation, both in simulation and on a real robot, when used within a model predictive controller.