# Model-free vision-based shaping of deformable plastic materials

arXiv: https://arxiv.org/abs/2001.11196 · HTML: https://mlsanity.com/paper/2001.11196/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2020.

## Built on by (most cited first, 5 of 5)

- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [Dynamic-Resolution Model Learning for Object Pile Manipulation](https://mlsanity.com/paper/2306.16700/) (2023)
- [Manipulation of Granular Materials by Learning Particle Interactions](https://mlsanity.com/paper/2111.02274/) (2021)
- [Differentiable Physics-based System Identification for Robotic Manipulation of Elastoplastic Materials](https://mlsanity.com/paper/2411.00554/) (2024)
- [Interactive Shaping of Granular Media Using Reinforcement Learning](https://mlsanity.com/paper/2509.06469/) (2025)

## Abstract (arXiv)

We address the problem of shaping deformable plastic materials using non-prehensile actions. Shaping plastic objects is challenging, since they are difficult to model and to track visually. We study this problem, by using kinetic sand, a plastic toy material which mimics the physical properties of wet sand. Inspired by a pilot study where humans shape kinetic sand, we define two types of actions: textitpushing the material from the sides and textittapping from above. The chosen actions are executed with a robotic arm using image-based visual servoing. From the current and desired view of the material, we define states based on visual features such as the outer contour shape and the pixel luminosity values. These are mapped to actions, which are repeated iteratively to reduce the image error until convergence is reached. For pushing, we propose three methods for mapping the visual state to an action. These include heuristic methods and a neural network, trained from human actions. We show that it is possible to obtain simple shapes with the kinetic sand, without explicitly modeling the material. Our approach is limited in the types of shapes it can achieve. A richer set of action types and multi-step reasoning is needed to achieve more sophisticated shapes.