# Tactile-based Object Retrieval From Granular Media

arXiv: https://arxiv.org/abs/2402.04536 · HTML: https://mlsanity.com/paper/2402.04536/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 84 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation](https://mlsanity.com/paper/2303.02346/) (2023)
- [Improved GelSight Tactile Sensor for Measuring Geometry and Slip](https://mlsanity.com/paper/1708.00922/) (2017)
- [Inferring the Material Properties of Granular Media for Robotic Tasks](https://mlsanity.com/paper/2003.08032/) (2020)
- [Tactile SLAM: Real-time inference of shape and pose from planar pushing](https://mlsanity.com/paper/2011.07044/) (2020)
- [Deep Reinforcement Learning for Robotic Pushing and Picking in Cluttered Environment](https://mlsanity.com/paper/2302.10717/) (2023)
- [TANDEM: Learning Joint Exploration and Decision Making with Tactile Sensors](https://mlsanity.com/paper/2203.00798/) (2022)
- [SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks](https://mlsanity.com/paper/2003.04956/) (2020)
- [TANDEM3D: Active Tactile Exploration for 3D Object Recognition](https://mlsanity.com/paper/2209.08772/) (2022)
- [A Sensorized Multicurved Robot Finger with Data-driven Touch Sensing via Overlapping Light Signals](https://mlsanity.com/paper/2004.00685/) (2020)
- [Digger Finger: GelSight Tactile Sensor for Object Identification Inside Granular Media](https://mlsanity.com/paper/2102.10230/) (2021)
- [Manipulation of Granular Materials by Learning Particle Interactions](https://mlsanity.com/paper/2111.02274/) (2021)
- [Touch-based Curiosity for Sparse-Reward Tasks](https://mlsanity.com/paper/2104.00442/) (2021)
- [Few-shot Adaptation for Manipulating Granular Materials Under Domain Shift](https://mlsanity.com/paper/2303.02893/) (2023)
- [A multiscale model of terrain dynamics for real-time earthmoving simulation](https://mlsanity.com/paper/2011.00459/) (2020)
- [Active End-Effector Pose Selection for Tactile Object Recognition through Monte Carlo Tree Search](https://mlsanity.com/paper/1703.00095/) (2017)
- [Granular Gym: High Performance Simulation for Robotic Tasks with Granular Materials](https://mlsanity.com/paper/2306.01369/) (2023)
- [GRAINS: Proximity Sensing of Objects in Granular Materials](https://mlsanity.com/paper/2307.05935/) (2023)
- [An Investigation of Multi-feature Extraction and Super-resolution with Fast Microphone Arrays](https://mlsanity.com/paper/2310.00206/) (2023)

## Built on by (most cited first, 5 of 5)

- [ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data](https://mlsanity.com/paper/2406.19464/) (2024)
- [DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation](https://mlsanity.com/paper/2509.18830/) (2025)
- [Localized Graph-Based Neural Dynamics Models for Terrain Manipulation](https://mlsanity.com/paper/2503.23270/) (2025)
- [Interactive Shaping of Granular Media Using Reinforcement Learning](https://mlsanity.com/paper/2509.06469/) (2025)
- [Tactile Curiosity Drives Robot Interaction](https://mlsanity.com/paper/2609.40134/) (2026)

## Abstract (arXiv)

We introduce GEOTACT, the first robotic system capable of grasping and retrieving objects of potentially unknown shapes buried in a granular environment. While important in many applications, ranging from mining and exploration to search and rescue, this type of interaction with granular media is difficult due to the uncertainty stemming from visual occlusion and noisy contact signals. To address these challenges, we use a learning method relying exclusively on touch feedback, trained end-to-end with simulated sensor noise. We show that our problem formulation leads to the natural emergence of learned pushing behaviors that the manipulator uses to reduce uncertainty and funnel the object to a stable grasp despite spurious and noisy tactile readings. We introduce a training curriculum that bootstraps learning in simulated granular environments, enabling zero-shot transfer to real hardware. Despite being trained only on seven objects with primitive shapes, our method is shown to successfully retrieve 35 different objects, including rigid, deformable, and articulated objects with complex shapes. Videos and additional information can be found at https://jxu.ai/geotact.