# Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

arXiv: https://arxiv.org/abs/2110.04685 · HTML: https://mlsanity.com/paper/2110.04685/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 10 Oct 2021.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Unsupervised Learning of Object Keypoints for Perception and Control](https://mlsanity.com/paper/1906.11883/) (2019)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning](https://mlsanity.com/paper/1810.05762/) (2018)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [Learning Predictive Representations for Deformable Objects Using Contrastive Estimation](https://mlsanity.com/paper/2003.05436/) (2020)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [Learning Continuous 3D Reconstructions for Geometrically Aware Grasping](https://mlsanity.com/paper/1910.00983/) (2019)
- [SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction](https://mlsanity.com/paper/2003.03472/) (2020)
- [Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery](https://mlsanity.com/paper/2109.02323/) (2021)
- [Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning](https://mlsanity.com/paper/2011.04813/) (2020)
- [Contrastive Variational Reinforcement Learning for Complex Observations](https://mlsanity.com/paper/2008.02430/) (2020)
- [Multi-Fingered Active Grasp Learning](https://mlsanity.com/paper/2006.05264/) (2020)
- [Non-Smooth Newton Methods for Deformable Multi-Body Dynamics](https://mlsanity.com/paper/1907.04587/) (2019)
- [Autonomous tissue retraction with a biomechanically informed logic based framework](https://mlsanity.com/paper/2109.02316/) (2021)
- [DefGraspSim: Simulation-based grasping of 3D deformable objects](https://mlsanity.com/paper/2107.05778/) (2021)
- [Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contours](https://mlsanity.com/paper/2006.09023/) (2020)
- [Contour Moments Based Manipulation of Composite Rigid-Deformable Objects with Finite Time Model Estimation and Shape/Position Control](https://mlsanity.com/paper/2106.02424/) (2021)
- [Bandit-Based Model Selection for Deformable Object Manipulation](https://mlsanity.com/paper/1703.10254/) (2017)
- [DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation](https://mlsanity.com/paper/2107.08067/) (2021)

## Built on by (most cited first, 9 of 9)

- [Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2312.10008/) (2023)
- [SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy](https://mlsanity.com/paper/2403.10401/) (2024)
- [DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects](https://mlsanity.com/paper/2305.04449/) (2023)
- [LUDO: Low-Latency Understanding of Deformable Objects using Point Cloud Occupancy Functions](https://mlsanity.com/paper/2411.08777/) (2024)
- [DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulation](https://mlsanity.com/paper/2309.14463/) (2023)
- [SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data](https://mlsanity.com/paper/2309.13863/) (2023)
- [Implicit Neural-Representation Learning for Elastic Deformable-Object Manipulations](https://mlsanity.com/paper/2505.00500/) (2025)
- [Learning visual-based deformable object rearrangement with local graph neural networks](https://mlsanity.com/paper/2310.10307/) (2023)
- [DiffDef: A Diffusion Model for Generating Multimodal Goal Shapes From Demonstrations for Deformable Object Manipulation](https://mlsanity.com/paper/2506.18779/) (2025)

## Abstract (arXiv)

If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the object being manipulated and a point cloud of the goal shape to learn a low-dimensional representation of the object shape. This shape embedding enables the robot to learn to define a visual servo controller that provides Cartesian pose changes to the robot end-effector causing the object to deform towards its target shape. Crucially, we demonstrate both in simulation and on a physical robot that DeformerNet reliably generalizes to object shapes and material stiffness not seen during training and outperforms comparison methods for both the generic shape control and the surgical task of retraction.