# Manipulating Deformable Objects by Interleaving Prediction, Planning, and Control

arXiv: https://arxiv.org/abs/2001.09950 · HTML: https://mlsanity.com/paper/2001.09950/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2020.

## Built on by (most cited first, 9 of 9)

- [Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics](https://mlsanity.com/paper/2205.02835/) (2022)
- [Deep Reinforcement Learning Based on Local GNN for Goal-conditioned Deformable Object Rearranging](https://mlsanity.com/paper/2302.10446/) (2023)
- [Differentiable Physics-based System Identification for Robotic Manipulation of Elastoplastic Materials](https://mlsanity.com/paper/2411.00554/) (2024)
- [Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulation](https://mlsanity.com/paper/2606.14188/) (2026)
- [Graph-Transporter: A Graph-based Learning Method for Goal-Conditioned Deformable Object Rearranging Task](https://mlsanity.com/paper/2302.10445/) (2023)
- [Robotic Dough Shaping](https://mlsanity.com/paper/2208.00386/) (2022)
- [Certifiably Safe Manipulation of Deformable Linear Objects via Joint Shape and Tension Prediction](https://mlsanity.com/paper/2505.13889/) (2025)
- [Co-manipulation of soft-materials estimating deformation from depth images](https://mlsanity.com/paper/2301.05609/) (2023)
- [Avoidance of an unexpected obstacle without reinforcement learning: Why not using advanced control-theoretic tools?](https://mlsanity.com/paper/2509.03721/) (2025)

## Abstract (arXiv)

We present a framework for deformable object manipulation that interleaves planning and control, enabling complex manipulation tasks without relying on high-fidelity modeling or simulation. The key question we address is when should we use planning and when should we use control to achieve the task? Planners are designed to find paths through complex configuration spaces, but for highly underactuated systems, such as deformable objects, achieving a specific configuration is very difficult even with high-fidelity models. Conversely, controllers can be designed to achieve specific configurations, but they can be trapped in undesirable local minima due to obstacles. Our approach consists of three components: (1) A global motion planner to generate gross motion of the deformable object; (2) A local controller for refinement of the configuration of the deformable object; and (3) A novel deadlock prediction algorithm to determine when to use planning versus control. By separating planning from control we are able to use different representations of the deformable object, reducing overall complexity and enabling efficient computation of motion. We provide a detailed proof of probabilistic completeness for our planner, which is valid despite the fact that our system is underactuated and we do not have a steering function. We then demonstrate that our framework is able to successfully perform several manipulation tasks with rope and cloth in simulation which cannot be performed using either our controller or planner alone. These experiments suggest that our planner can generate paths efficiently, taking under a second on average to find a feasible path in three out of four scenarios. We also show that our framework is effective on a 16 DoF physical robot, where reachability and dual-arm constraints make the planning more difficult.