# DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulation

arXiv: https://arxiv.org/abs/2309.14463 · HTML: https://mlsanity.com/paper/2309.14463/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions](https://mlsanity.com/paper/1905.06292/) (2019)
- [Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction](https://mlsanity.com/paper/1706.07036/) (2017)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning](https://mlsanity.com/paper/1810.05762/) (2018)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [C-Flow: Conditional Generative Flow Models for Images and 3D Point Clouds](https://mlsanity.com/paper/1912.07009/) (2019)
- [Challenges and Outlook in Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2105.01767/) (2021)
- [Learning Continuous 3D Reconstructions for Geometrically Aware Grasping](https://mlsanity.com/paper/1910.00983/) (2019)
- [SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction](https://mlsanity.com/paper/2003.03472/) (2020)
- [Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery](https://mlsanity.com/paper/2109.02323/) (2021)
- [Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning](https://mlsanity.com/paper/2011.04813/) (2020)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)
- [Learning Local Displacements for Point Cloud Completion](https://mlsanity.com/paper/2203.16600/) (2022)
- [Multi-Fingered Active Grasp Learning](https://mlsanity.com/paper/2006.05264/) (2020)
- [A Progressive Conditional Generative Adversarial Network for Generating Dense and Colored 3D Point Clouds](https://mlsanity.com/paper/2010.05391/) (2020)
- [Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds](https://mlsanity.com/paper/2110.04685/) (2021)
- [Autonomously Navigating a Surgical Tool Inside the Eye by Learning from Demonstration](https://mlsanity.com/paper/2011.07785/) (2020)
- [Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contours](https://mlsanity.com/paper/2006.09023/) (2020)
- [Lattice-based shape tracking and servoing of elastic objects](https://mlsanity.com/paper/2209.01832/) (2022)
- [Contour Moments Based Manipulation of Composite Rigid-Deformable Objects with Finite Time Model Estimation and Shape/Position Control](https://mlsanity.com/paper/2106.02424/) (2021)
- [DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects](https://mlsanity.com/paper/2305.04449/) (2023)
- [DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation](https://mlsanity.com/paper/2107.08067/) (2021)
- [Model Predictive Manipulation of Compliant Objects with Multi-Objective Optimizer and Adversarial Network for Occlusion Compensation](https://mlsanity.com/paper/2205.09987/) (2022)

## Built on by (most cited first, 3 of 3)

- [Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control](https://mlsanity.com/paper/2505.16249/) (2025)
- [DiffDef: A Diffusion Model for Generating Multimodal Goal Shapes From Demonstrations for Deformable Object Manipulation](https://mlsanity.com/paper/2506.18779/) (2025)
- [Few-Shot Adaptation to Non-Stationary Environments via Latent Trend Embedding for Robotics](https://mlsanity.com/paper/2603.10373/) (2026)

## Abstract (arXiv)

Shape servoing, a robotic task dedicated to controlling objects to desired goal shapes, is a promising approach to deformable object manipulation. An issue arises, however, with the reliance on the specification of a goal shape. This goal has been obtained either by a laborious domain knowledge engineering process or by manually manipulating the object into the desired shape and capturing the goal shape at that specific moment, both of which are impractical in various robotic applications. In this paper, we solve this problem by developing a novel neural network DefGoalNet, which learns deformable object goal shapes directly from a small number of human demonstrations. We demonstrate our method's effectiveness on various robotic tasks, both in simulation and on a physical robot. Notably, in the surgical retraction task, even when trained with as few as 10 demonstrations, our method achieves a median success percentage of nearly 90%. These results mark a substantial advancement in enabling shape servoing methods to bring deformable object manipulation closer to practical, real-world applications.