mlsanityData to 14 Oct 2026

SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation

arXiv 2309.08728 ↗First posted 15 Sep 2023cs.RO cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Deformable object manipulation presents a unique set of challenges in robotic manipulation by exhibiting high degrees of freedom and severe self-occlusion. State representation for materials that exhibit plastic behavior, like modeling clay or bread dough, is also difficult because they permanently deform under stress and are constantly changing shape. In this work, we investigate each of these challenges using the task of robotic sculpting with a parallel gripper. We propose a system that uses point clouds as the state representation and leverages pre-trained point cloud reconstruction Transformer to learn a latent dynamics model to predict material deformations given a grasp action. We design a novel action sampling algorithm that reasons about geometrical differences between point clouds to further improve the efficiency of model-based planners. All data and experiments are conducted entirely in the real world. Our experiments show the proposed system is able to successfully capture the dynamics of clay, and is able to create a variety of simple shapes.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

35 of 55 references.

References · 35

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2020Point Transformercited by 1,470
  7. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  8. 2020Mastering Atari with Discrete World Modelscited by 1,007
  9. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  10. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  11. 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
  12. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
23 more
  1. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  2. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 131
  3. 2022Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgerycited by 123
  4. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 121
  5. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 103
  6. 2021FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfoldingcited by 81
  7. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  8. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  9. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 72
  10. 2019SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Roboticscited by 47
  11. 2019Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objectscited by 45
  12. 2019Dynamic Cloth Manipulation with Deep Reinforcement Learningcited by 29
  13. 2022VIRDO: Visio-tactile Implicit Representations of Deformable Objectscited by 25
  14. 2022Visuotactile Affordances for Cloth Manipulation with Local Controlcited by 21
  15. 2015Folding Deformable Objects using Predictive Simulation and Trajectory Optimizationcited by 19
  16. 2019A Grasping-centered Analysis for Cloth Manipulationcited by 15
  17. 2020Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics for Surgical Robot Autonomycited by 11
  18. 2022EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamicscited by 10
  19. 2023Self-supervised Cloth Reconstruction via Action-conditioned Cloth Trackingcited by 9
  20. 2023OpenVR: Teleoperation for Manipulationcited by 8
  21. 2021Deformable Linear Object Prediction Using Locally Linear Latent Dynamicscited by 7
  22. 2022A Coarse-to-Fine Framework for Dual-Arm Manipulation of Deformable Linear Objects with Whole-Body Obstacle Avoidancecited by 5
  23. 2023RoboChop: Autonomous Framework for Fruit and Vegetable Chopping Leveraging Foundational Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 1.

8 papers.

  1. 2024VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policiescited by 15
  2. 2024DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objectscited by 9
  3. 2024SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policycited by 7
  4. 2024PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Roboticscited by 4
  5. 2024DOFS: A Real-world 3D Deformable Object Dataset with Full Spatial Information for Dynamics Model Learningcited by 3
  6. 2025Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Controlcited by 2
  7. 2025PinchBot: Long-Horizon Deformable Manipulation with Guided Diffusion Policycited by 1
  8. 2026Visual Sculpting: Visually-Aligned Planning Representations for Long-Horizon Robot Clay Sculptingcited by 0