mlsanityData to 7 Oct 2026

Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

arXiv 2110.04685 ↗First posted 10 Oct 2021cs.RO cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the object being manipulated and a point cloud of the goal shape to learn a low-dimensional representation of the object shape. This shape embedding enables the robot to learn to define a visual servo controller that provides Cartesian pose changes to the robot end-effector causing the object to deform towards its target shape. Crucially, we demonstrate both in simulation and on a physical robot that DeformerNet reliably generalizes to object shapes and material stiffness not seen during training and outperforms comparison methods for both the generic shape control and the surgical task of retraction.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 10 Oct 2021.

Builds on

28 of 49 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2018Group Normalizationcited by 1,514
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  6. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  7. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  8. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  9. 2019Self-supervised 6D Object Pose Estimation for Robot Manipulationcited by 100
  10. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  11. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 77
  12. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
16 more
  1. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  2. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  3. 2019Learning Continuous 3D Reconstructions for Geometrically Aware Graspingcited by 28
  4. 2020SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extractioncited by 28
  5. 2021SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learningcited by 28
  6. 2021Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgerycited by 22
  7. 2020Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planningcited by 19
  8. 2020Contrastive Variational Reinforcement Learning for Complex Observationscited by 15
  9. 2020Multi-Fingered Active Grasp Learningcited by 13
  10. 2019Non-Smooth Newton Methods for Deformable Multi-Body Dynamicscited by 7
  11. 2021DefGraspSim: Simulation-based grasping of 3D deformable objectscited by 6
  12. 2021Autonomous tissue retraction with a biomechanically informed logic based frameworkcited by 6
  13. 2020Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contourscited by 5
  14. 2021Contour Moments Based Manipulation of Composite Rigid-Deformable Objects with Finite Time Model Estimation and Shape/Position Controlcited by 4
  15. 2017Bandit-Based Model Selection for Deformable Object Manipulationcited by 3
  16. 2021DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 2; 2025: 2.

9 papers.

  1. 2023Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objectscited by 40
  2. 2024SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policycited by 7
  3. 2023DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objectscited by 4
  4. 2024LUDO: Low-Latency Understanding of Deformable Objects using Point Cloud Occupancy Functionscited by 4
  5. 2023DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulationcited by 3
  6. 2023SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Datacited by 2
  7. 2025Implicit Neural-Representation Learning for Elastic Deformable-Object Manipulationscited by 2
  8. 2023Learning visual-based deformable object rearrangement with local graph neural networkscited by 1
  9. 2025DiffDef: A Diffusion Model for Generating Multimodal Goal Shapes From Demonstrations for Deformable Object Manipulationcited by 0