# Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

arXiv: https://arxiv.org/abs/2003.05436 · HTML: https://mlsanity.com/paper/2003.05436/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2020.

## Built on by (most cited first, 30 of 72)

- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [Learning Visible Connectivity Dynamics for Cloth Smoothing](https://mlsanity.com/paper/2105.10389/) (2021)
- [Learning Task Informed Abstractions](https://mlsanity.com/paper/2106.15612/) (2021)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy](https://mlsanity.com/paper/2111.05623/) (2021)
- [Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation](https://mlsanity.com/paper/2201.07779/) (2022)
- [Visual Reinforcement Learning with Self-Supervised 3D Representations](https://mlsanity.com/paper/2210.07241/) (2022)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining](https://mlsanity.com/paper/2204.02393/) (2022)
- [D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement](https://mlsanity.com/paper/2309.16118/) (2023)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [DreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction](https://mlsanity.com/paper/2203.00494/) (2022)
- [GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy](https://mlsanity.com/paper/2410.17488/) (2024)
- [ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation](https://mlsanity.com/paper/2203.06856/) (2022)
- [Learning World Models with Identifiable Factorization](https://mlsanity.com/paper/2306.06561/) (2023)
- [Visuotactile Affordances for Cloth Manipulation with Local Control](https://mlsanity.com/paper/2212.05108/) (2022)
- [Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulation](https://mlsanity.com/paper/2503.11999/) (2025)
- [VisuoSpatial Foresight for Physical Sequential Fabric Manipulation](https://mlsanity.com/paper/2102.09754/) (2021)
- [Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects](https://mlsanity.com/paper/2104.12149/) (2021)
- [Benchmarking the Sim-to-Real Gap in Cloth Manipulation](https://mlsanity.com/paper/2310.09543/) (2023)
- [Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies](https://mlsanity.com/paper/2107.08942/) (2021)
- [One After Another: Learning Incremental Skills for a Changing World](https://mlsanity.com/paper/2203.11176/) (2022)
- [EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamics](https://mlsanity.com/paper/2209.08996/) (2022)
- [Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention](https://mlsanity.com/paper/2301.03003/) (2023)
- [Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds](https://mlsanity.com/paper/2110.04685/) (2021)
- [Autonomously Untangling Long Cables](https://mlsanity.com/paper/2207.07813/) (2022)

## Abstract (arXiv)

Using visual model-based learning for deformable object manipulation is challenging due to difficulties in learning plannable visual representations along with complex dynamic models. In this work, we propose a new learning framework that jointly optimizes both the visual representation model and the dynamics model using contrastive estimation. Using simulation data collected by randomly perturbing deformable objects on a table, we learn latent dynamics models for these objects in an offline fashion. Then, using the learned models, we use simple model-based planning to solve challenging deformable object manipulation tasks such as spreading ropes and cloths. Experimentally, we show substantial improvements in performance over standard model-based learning techniques across our rope and cloth manipulation suite. Finally, we transfer our visual manipulation policies trained on data purely collected in simulation to a real PR2 robot through domain randomization.