# DiffDef: A Diffusion Model for Generating Multimodal Goal Shapes From Demonstrations for Deformable Object Manipulation

arXiv: https://arxiv.org/abs/2506.18779 · HTML: https://mlsanity.com/paper/2506.18779/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions](https://mlsanity.com/paper/1905.06292/) (2019)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning](https://mlsanity.com/paper/1810.05762/) (2018)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [C-Flow: Conditional Generative Flow Models for Images and 3D Point Clouds](https://mlsanity.com/paper/1912.07009/) (2019)
- [Challenges and Outlook in Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2105.01767/) (2021)
- [Learning Continuous 3D Reconstructions for Geometrically Aware Grasping](https://mlsanity.com/paper/1910.00983/) (2019)
- [Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation](https://mlsanity.com/paper/2312.10807/) (2023)
- [Multi-Fingered Active Grasp Learning](https://mlsanity.com/paper/2006.05264/) (2020)
- [A Progressive Conditional Generative Adversarial Network for Generating Dense and Colored 3D Point Clouds](https://mlsanity.com/paper/2010.05391/) (2020)
- [Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds](https://mlsanity.com/paper/2110.04685/) (2021)
- [Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contours](https://mlsanity.com/paper/2006.09023/) (2020)
- [Lattice-based shape tracking and servoing of elastic objects](https://mlsanity.com/paper/2209.01832/) (2022)
- [DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects](https://mlsanity.com/paper/2305.04449/) (2023)
- [Model Predictive Manipulation of Compliant Objects with Multi-Objective Optimizer and Adversarial Network for Occlusion Compensation](https://mlsanity.com/paper/2205.09987/) (2022)
- [DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulation](https://mlsanity.com/paper/2309.14463/) (2023)
- [Stein Variational Belief Propagation for Multi-Robot Coordination](https://mlsanity.com/paper/2311.16916/) (2023)

## Abstract (arXiv)

Deformable object manipulation is a key capability in many robotic applications. A promising paradigm for this problem is shape servoing, which aims to control deformable objects toward desired goal shapes. However, existing approaches typically rely on impractical goal-shape acquisition methods, such as domain-knowledge engineering or manual manipulation. Moreover, prior methods generally assume a single deterministic goal and fail to handle multimodal goal settings, a common scenario in many real-world tasks where multiple distinct goal shapes can all lead to successful task completion. In this paper, we introduce DiffDef, a novel neural network that uses a diffusion model to learn a distribution of feasible goal shapes rather than predicting a single deterministic outcome. This allows DiffDef to generate diverse goal configurations while avoiding the mode-averaging artifacts common in deterministic predictors. We evaluate our method on several deformable manipulation tasks inspired by manufacturing and surgical applications, both in simulation and on two physical robotic platforms: the da Vinci Research Kit (dVRK) and a bimanual KUKA-based robotic system. The results demonstrate that DiffDef effectively captures multimodal goal distributions and significantly improves task performance in practical robotic settings. Website: sites.google.com/view/diffdef.