# Visual Sculpting: Visually-Aligned Planning Representations for Long-Horizon Robot Clay Sculpting

arXiv: https://arxiv.org/abs/2605.17556 · HTML: https://mlsanity.com/paper/2605.17556/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 17 May 2026.

## Builds on (most cited first, 11 of 11)

- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools](https://mlsanity.com/paper/2306.14447/) (2023)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [FRIDA: A Collaborative Robot Painter with a Differentiable, Real2Sim2Real Planning Environment](https://mlsanity.com/paper/2210.00664/) (2022)
- [SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation](https://mlsanity.com/paper/2309.08728/) (2023)
- [SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy](https://mlsanity.com/paper/2403.10401/) (2024)
- [LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models](https://mlsanity.com/paper/2406.08648/) (2024)
- [RoPotter: Toward Robotic Pottery and Deformable Object Manipulation with Structural Priors](https://mlsanity.com/paper/2408.02184/) (2024)
- [Planning and Reasoning with 3D Deformable Objects for Hierarchical Text-to-3D Robotic Shaping](https://mlsanity.com/paper/2412.01765/) (2024)

## Abstract (arXiv)

Clay sculpting is a nuanced, artistic task involving dexterous manipulation with long-horizon planning to achieve high-level goals. As a robotics problem, we formulate clay sculpting as a shape-to-shape matching challenge. Prior deformable object manipulation work either requires retraining a policy per goal or relies on dynamics models which represent state as sparse point clouds which do not capture important clay features, such as textures, well. We present a method for modeling the dynamics of deformable materials and planning for robotic sculpting in a representation that is visually-aligned, capturing lighting and texture features. With three different deformable materials and various end-effectors, we demonstrate that our dynamics model is comparable in performance to the state-of-the-art with the added benefit of being compatible with visual planning. Our actions are represented as parametrized pushes into clay with a single end-effector, which proved to be suitable for long-horizon (>100 actions) clay relief sculptures. Lastly, we show the benefits of planning in a visually-aligned representation, but also provide analysis providing evidence as to why this representation is challenging to plan in compared to 3D representations.