# Learning Language-Conditioned Deformable Object Manipulation with Graph Dynamics

arXiv: https://arxiv.org/abs/2303.01310 · HTML: https://mlsanity.com/paper/2303.01310/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2023.

## Built on by (most cited first, 11 of 11)

- [Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision](https://mlsanity.com/paper/2501.01715/) (2025)
- [Graph learning in robotics: a survey](https://mlsanity.com/paper/2310.04294/) (2023)
- [TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach](https://mlsanity.com/paper/2407.03245/) (2024)
- [BiFold: Bimanual Cloth Folding with Language Guidance](https://mlsanity.com/paper/2501.16458/) (2025)
- [GenORM: Generalizable One-shot Rope Manipulation with Parameter-Aware Policy](https://mlsanity.com/paper/2306.09872/) (2023)
- [Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions](https://mlsanity.com/paper/2506.03760/) (2025)
- [CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints](https://mlsanity.com/paper/2507.19983/) (2025)
- [Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Folding](https://mlsanity.com/paper/2505.07600/) (2025)
- [Learning-based Cooperative Robotic Paper Wrapping: A Unified Control Policy with Residual Force Control](https://mlsanity.com/paper/2511.03181/) (2025)
- [Vision-Based 6-DoF Grasp Pose Estimation for Robot Cloth Unfolding](https://mlsanity.com/paper/2609.31452/) (2026)
- [NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fields](https://mlsanity.com/paper/2610.00981/) (2026)

## Abstract (arXiv)

Multi-task learning of deformable object manipulation is a challenging problem in robot manipulation. Most previous works address this problem in a goal-conditioned way and adapt goal images to specify different tasks, which limits the multi-task learning performance and can not generalize to new tasks. Thus, we adapt language instruction to specify deformable object manipulation tasks and propose a learning framework. We first design a unified Transformer-based architecture to understand multi-modal data and output picking and placing action. Besides, we have introduced the visible connectivity graph to tackle nonlinear dynamics and complex configuration of the deformable object. Both simulated and real experiments have demonstrated that the proposed method is effective and can generalize to unseen instructions and tasks. Compared with the state-of-the-art method, our method achieves higher success rates (87.2% on average) and has a 75.6% shorter inference time. We also demonstrate that our method performs well in real-world experiments.