mlsanityData to 7 Oct 2026

SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy

arXiv 2403.10401 ↗First posted 15 Mar 2024cs.RO cs.AIcited by 7

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Manipulating deformable objects remains a challenge within robotics due to the difficulties of state estimation, long-horizon planning, and predicting how the object will deform given an interaction. These challenges are the most pronounced with 3D deformable objects. We propose SculptDiff, a goal-conditioned diffusion-based imitation learning framework that works with point cloud state observations to directly learn clay sculpting policies for a variety of target shapes. To the best of our knowledge this is the first real-world method that successfully learns manipulation policies for 3D deformable objects. For sculpting videos and access to our dataset and hardware CAD models, see the project website: https://sites.google.com/andrew.cmu.edu/imitation-sculpting/home

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 33 of its 44 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

33 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  5. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  8. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  9. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  10. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  11. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  12. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
20 more
  1. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 75
  2. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  3. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  4. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  5. 2021FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policycited by 54
  6. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  7. 2022DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Toolscited by 44
  8. 2022Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulationcited by 32
  9. 2022ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulationcited by 24
  10. 2023DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physicscited by 23
  11. 2023One ACT Play: Single Demonstration Behavior Cloning with Action Chunking Transformerscited by 17
  12. 2022Contact Points Discovery for Soft-Body Manipulations with Differentiable Physicscited by 14
  13. 2020Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics for Surgical Robot Autonomycited by 11
  14. 2022Learning Closed-loop Dough Manipulation Using a Differentiable Reset Modulecited by 11
  15. 2021Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Cloudscited by 9
  16. 2023SculptBot: Pre-Trained Models for 3D Deformable Object Manipulationcited by 8
  17. 2021Deformable Linear Object Prediction Using Locally Linear Latent Dynamicscited by 7
  18. 2023SoftGPT: Learn Goal-oriented Soft Object Manipulation Skills by Generative Pre-trained Heterogeneous Graph Transformercited by 7
  19. 2022A Coarse-to-Fine Framework for Dual-Arm Manipulation of Deformable Linear Objects with Whole-Body Obstacle Avoidancecited by 4
  20. 2021DiffSRL: Learning Dynamical State Representation for Deformable Object Manipulation with Differentiable Simulatorcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025RT-Cache: Training-Free Retrieval for Real-Time Manipulationcited by 5
  2. 2025Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Controlcited by 2
  3. 2025VLM-TDP: VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulationcited by 1
  4. 2025PinchBot: Long-Horizon Deformable Manipulation with Guided Diffusion Policycited by 1
  5. 2026LinkD: AutoRegressive Diffusion Model for Mechanical Linkage Synthesiscited by 0
  6. 2026Visual Sculpting: Visually-Aligned Planning Representations for Long-Horizon Robot Clay Sculptingcited by 0
  7. 2026Rotation-Aware Point-Cloud Embeddings for Vision-Based In-Hand Reorientationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Planning and Reasoning with 3D Deformable Objects for Hierarchical Text-to-3D Robotic Shaping
  2. 2026Online Material Estimation for Conditioned Diffusion Policy in Shaping Deformable Linear Objects
  3. 2025DiffDef: A Diffusion Model for Generating Multimodal Goal Shapes From Demonstrations for Deformable Object Manipulation
  4. 2022ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds
  5. 2025Dynamic Manipulation of Deformable Objects in 3D: Simulation, Benchmark and Learning Strategy
  6. 2022Learning Deformable Object Manipulation from Expert Demonstrations
  7. 2024Learning Generalizable 3D Manipulation With 10 Demonstrations
  8. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulation