mlsanityData to 7 Oct 2026

DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

arXiv 2305.04449 ↗First posted 8 May 2023cs.RO cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Applications in fields ranging from home care to warehouse fulfillment to surgical assistance require robots to reliably manipulate the shape of 3D deformable objects. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the manipulated object and a point cloud of the goal shape to learn a low-dimensional representation of the object shape. This shape embedding enables the robot to learn a visual servo controller that computes the desired robot end-effector action to iteratively deform the object toward the target shape. We demonstrate both in simulation and on a physical robot that DeformerNet reliably generalizes to object shapes and material stiffness not seen during training, including ex vivo chicken muscle tissue. Crucially, using DeformerNet, the robot successfully accomplishes three surgical sub-tasks: retraction (moving tissue aside to access a site underneath it), tissue wrapping (a sub-task in procedures like aortic stent placements), and connecting two tubular pieces of tissue (a sub-task in anastomosis).

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 71 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

43 of 71 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2018Group Normalizationcited by 1,514
  5. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  8. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  9. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  10. 2019Self-supervised 6D Object Pose Estimation for Robot Manipulationcited by 100
  11. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  12. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 77
31 more
  1. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  2. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  3. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  4. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  5. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  6. 2021Challenges and Outlook in Robotic Manipulation of Deformable Objectscited by 43
  7. 2019Learning Continuous 3D Reconstructions for Geometrically Aware Graspingcited by 28
  8. 2020SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extractioncited by 28
  9. 2021SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learningcited by 28
  10. 2022ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulationcited by 24
  11. 2021Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgerycited by 22
  12. 2020Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planningcited by 19
  13. 2022Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Trackingcited by 19
  14. 2020Contrastive Variational Reinforcement Learning for Complex Observationscited by 15
  15. 2020Multi-Fingered Active Grasp Learningcited by 13
  16. 2020Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulationcited by 13
  17. 2022VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objectscited by 13
  18. 2018Densely Supervised Grasp Detector (DSGD)cited by 10
  19. 2021Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Cloudscited by 9
  20. 2020Autonomously Navigating a Surgical Tool Inside the Eye by Learning from Demonstrationcited by 8
  21. 2019Non-Smooth Newton Methods for Deformable Multi-Body Dynamicscited by 7
  22. 2021DefGraspSim: Simulation-based grasping of 3D deformable objectscited by 6
  23. 2021Autonomous tissue retraction with a biomechanically informed logic based frameworkcited by 6
  24. 2020Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contourscited by 5
  25. 2022Lattice-based shape tracking and servoing of elastic objectscited by 5
  26. 2021Contour Moments Based Manipulation of Composite Rigid-Deformable Objects with Finite Time Model Estimation and Shape/Position Controlcited by 4
  27. 2017Bandit-Based Model Selection for Deformable Object Manipulationcited by 3
  28. 2019Toward Synergic Learning for Autonomous Manipulation of Deformable Tissues via Surgical Robots: An Approximate Q-Learning Approachcited by 3
  29. 2021DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulationcited by 3
  30. 2022Model Predictive Manipulation of Compliant Objects with Multi-Objective Optimizer and Adversarial Network for Occlusion Compensationcited by 3
  31. 2021Toward Learning Context-Dependent Tasks from Demonstration for Tendon-Driven Surgical Robotscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2024Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulationscited by 20
  2. 2023DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulationcited by 3
  3. 2025Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Controlcited by 2
  4. 2025DiffDef: A Diffusion Model for Generating Multimodal Goal Shapes From Demonstrations for Deformable Object Manipulationcited by 0