# Deformable Linear Object Prediction Using Locally Linear Latent Dynamics

arXiv: https://arxiv.org/abs/2103.14184 · HTML: https://mlsanity.com/paper/2103.14184/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1708.06850/) (2017)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [Learning Compositional Koopman Operators for Model-Based Control](https://mlsanity.com/paper/1910.08264/) (2019)
- [Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data](https://mlsanity.com/paper/2003.01835/) (2020)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects](https://mlsanity.com/paper/1911.06283/) (2019)
- [Learning Robotic Manipulation through Visual Planning and Acting](https://mlsanity.com/paper/1905.04411/) (2019)
- [Model-Driven Feed-Forward Prediction for Manipulation of Deformable Objects](https://mlsanity.com/paper/1607.04411/) (2016)

## Built on by (most cited first, 7 of 7)

- [Closing the Train-Test Gap in World Models for Gradient-Based Planning](https://mlsanity.com/paper/2512.09929/) (2025)
- [SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation](https://mlsanity.com/paper/2309.08728/) (2023)
- [Disentangling Dense Multi-Cable Knots](https://mlsanity.com/paper/2106.02252/) (2021)
- [SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy](https://mlsanity.com/paper/2403.10401/) (2024)
- [Representation Learning for Continuous Action Spaces is Beneficial for Efficient Policy Learning](https://mlsanity.com/paper/2211.13257/) (2022)
- [Learning Quasi-Static 3D Models of Markerless Deformable Linear Objects for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2309.07609/) (2023)
- [DLO-Splatting: Tracking Deformable Linear Objects Using 3D Gaussian Splatting](https://mlsanity.com/paper/2505.08644/) (2025)

## Abstract (arXiv)

We propose a framework for deformable linear object prediction. Prediction of deformable objects (e.g., rope) is challenging due to their non-linear dynamics and infinite-dimensional configuration spaces. By mapping the dynamics from a non-linear space to a linear space, we can use the good properties of linear dynamics for easier learning and more efficient prediction. We learn a locally linear, action-conditioned dynamics model that can be used to predict future latent states. Then, we decode the predicted latent state into the predicted state. We also apply a sampling-based optimization algorithm to select the optimal control action. We empirically demonstrate that our approach can predict the rope state accurately up to ten steps into the future and that our algorithm can find the optimal action given an initial state and a goal state.