# Learning Gentle Grasping Using Vision, Sound, and Touch

arXiv: https://arxiv.org/abs/2503.07926 · HTML: https://mlsanity.com/paper/2503.07926/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [An Exponential Learning Rate Schedule for Deep Learning](https://mlsanity.com/paper/1910.07454/) (2019)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017)
- [ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data](https://mlsanity.com/paper/2406.19464/) (2024)
- [Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning](https://mlsanity.com/paper/1903.08542/) (2019)
- [Digitizing Touch with an Artificial Multimodal Fingertip](https://mlsanity.com/paper/2411.02479/) (2024)
- [Towards Learning to Play Piano with Dexterous Hands and Touch](https://mlsanity.com/paper/2106.02040/) (2021)
- [Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer](https://mlsanity.com/paper/2112.06374/) (2021)
- [Tactile-Driven Gentle Grasping for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/2303.09346/) (2023)
- [A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval](https://mlsanity.com/paper/2407.20709/) (2024)

## Built on by (most cited first, 2 of 2)

- [Learning to Play Piano in the Real World](https://mlsanity.com/paper/2503.15481/) (2025)
- [Calibration-Free Surface Normals Estimation in Vision-Based Tactile Sensing using Universal Photometric Stereo](https://mlsanity.com/paper/2609.31754/) (2026)

## Abstract (arXiv)

In our daily life, we often encounter objects that are fragile and can be damaged by excessive grasping force, such as fruits. For these objects, it is paramount to grasp gently -- not using the maximum amount of force possible, but rather the minimum amount of force necessary. This paper proposes using visual, tactile, and auditory signals to learn to grasp and regrasp objects stably and gently. Specifically, we use audio signals as an indicator of gentleness during the grasping, and then train an end-to-end action-conditional model from raw visuo-tactile inputs that predicts both the stability and the gentleness of future grasping candidates, thus allowing the selection and execution of the most promising action. Experimental results on a multi-fingered hand over 1,500 grasping trials demonstrated that our model is useful for gentle grasping by validating the predictive performance (3.27% higher accuracy than the vision-only variant) and providing interpretations of their behavior. Finally, real-world experiments confirmed that the grasping performance with the trained multi-modal model outperformed other baselines (17% higher rate for stable and gentle grasps than vision-only). Our approach requires neither tactile sensor calibration nor analytical force modeling, drastically reducing the engineering effort to grasp fragile objects. Dataset and videos are available at https://lasr.org/research/gentle-grasping.