# Interactive Shaping of Granular Media Using Reinforcement Learning

arXiv: https://arxiv.org/abs/2509.06469 · HTML: https://mlsanity.com/paper/2509.06469/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 25 of 25)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Training](https://mlsanity.com/paper/2305.12127/) (2023)
- [Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving](https://mlsanity.com/paper/2410.24010/) (2024)
- [Learning Robotic Manipulation of Granular Media](https://mlsanity.com/paper/1709.02833/) (2017)
- [Manipulation of Granular Materials by Learning Particle Interactions](https://mlsanity.com/paper/2111.02274/) (2021)
- [Excavation Reinforcement Learning Using Geometric Representation](https://mlsanity.com/paper/2201.11292/) (2022)
- [Neural Field Dynamics Model for Granular Object Piles Manipulation](https://mlsanity.com/paper/2311.00802/) (2023)
- [HEAP -- The autonomous walking excavator](https://mlsanity.com/paper/2106.05059/) (2021)
- [Model-free vision-based shaping of deformable plastic materials](https://mlsanity.com/paper/2001.11196/) (2020)
- [Few-shot Adaptation for Manipulating Granular Materials Under Domain Shift](https://mlsanity.com/paper/2303.02893/) (2023)
- [Tactile-based Object Retrieval From Granular Media](https://mlsanity.com/paper/2402.04536/) (2024)
- [Developing a Simple Model for Sand-Tool Interaction and Autonomously Shaping Sand](https://mlsanity.com/paper/1908.02745/) (2019)
- [Learning Granular Media Avalanche Behavior for Indirectly Manipulating Obstacles on a Granular Slope](https://mlsanity.com/paper/2407.01898/) (2024)
- [Towards Autonomous Grading In The Real World](https://mlsanity.com/paper/2206.06091/) (2022)
- [Learning Excavation of Rigid Objects with Offline Reinforcement Learning](https://mlsanity.com/paper/2303.16427/) (2023)
- [Towards Autonomous Excavation Planning](https://mlsanity.com/paper/2308.11478/) (2023)
- [Localized Graph-Based Neural Dynamics Models for Terrain Manipulation](https://mlsanity.com/paper/2503.23270/) (2025)
- [Material-agnostic Shaping of Granular Materials with Optimal Transport](https://mlsanity.com/paper/2303.17047/) (2023)
- [Learning manipulation of steep granular slopes for fast Mini Rover turning](https://mlsanity.com/paper/2310.01273/) (2023)

## Built on by (most cited first, 1 of 1)

- [Size Doesn't Matter: Material-State Reinforcement Learning for Excavator Transferable Soil Manipulation](https://mlsanity.com/paper/2609.12677/) (2026)

## Abstract (arXiv)

Autonomous manipulation of granular media, such as sand, is crucial for applications in construction, excavation, and additive manufacturing. However, shaping granular materials presents unique challenges due to their high-dimensional configuration space and complex dynamics, where traditional rule-based approaches struggle without extensive engineering efforts. Reinforcement learning (RL) offers a promising alternative by enabling agents to learn adaptive manipulation strategies through trial and error. In this work, we present an RL framework that enables a robotic arm with a cubic end-effector and a stereo camera to shape granular media into desired target structures. We show the importance of compact observations and concise reward formulations for the large configuration space, validating our design choices with an ablation study. Our results demonstrate the effectiveness of the proposed approach for the training of visual policies that manipulate granular media including their real-world deployment, significantly outperforming two baseline approaches in terms of target shape accuracy.