# Seeing by haptic glance: reinforcement learning-based 3D object Recognition

arXiv: https://arxiv.org/abs/2102.07599 · HTML: https://mlsanity.com/paper/2102.07599/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 14 of 14)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Relation-Shape Convolutional Neural Network for Point Cloud Analysis](https://mlsanity.com/paper/1904.07601/) (2019)
- [PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention](https://mlsanity.com/paper/1810.05591/) (2018)
- [DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing](https://mlsanity.com/paper/1909.03669/) (2019)
- [Effective Rotation-invariant Point CNN with Spherical Harmonics kernels](https://mlsanity.com/paper/1906.11555/) (2019)
- [Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution](https://mlsanity.com/paper/1811.09361/) (2018)
- [RobustPointSet: A Dataset for Benchmarking Robustness of Point Cloud Classifiers](https://mlsanity.com/paper/2011.11572/) (2020)
- [Learning efficient haptic shape exploration with a rigid tactile sensor array](https://mlsanity.com/paper/1902.07501/) (2019)
- [PointMask: Towards Interpretable and Bias-Resilient Point Cloud Processing](https://mlsanity.com/paper/2007.04525/) (2020)

## Abstract (arXiv)

Human is able to conduct 3D recognition by a limited number of haptic contacts between the target object and his/her fingers without seeing the object. This capability is defined as `haptic glance' in cognitive neuroscience. Most of the existing 3D recognition models were developed based on dense 3D data. Nonetheless, in many real-life use cases, where robots are used to collect 3D data by haptic exploration, only a limited number of 3D points could be collected. In this study, we thus focus on solving the intractable problem of how to obtain cognitively representative 3D key-points of a target object with limited interactions between the robot and the object. A novel reinforcement learning based framework is proposed, where the haptic exploration procedure (the agent iteratively predicts the next position for the robot to explore) is optimized simultaneously with the objective 3D recognition with actively collected 3D points. As the model is rewarded only when the 3D object is accurately recognized, it is driven to find the sparse yet efficient haptic-perceptual 3D representation of the object. Experimental results show that our proposed model outperforms the state of the art models.