# Real-Time Tactile Grasp Force Sensing Using Fingernail Imaging via Deep Neural Networks

arXiv: https://arxiv.org/abs/2109.15231 · HTML: https://mlsanity.com/paper/2109.15231/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Socially and Contextually Aware Human Motion and Pose Forecasting](https://mlsanity.com/paper/2007.06843/) (2020)
- [11K Hands: Gender recognition and biometric identification using a large dataset of hand images](https://mlsanity.com/paper/1711.04322/) (2017)
- [Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction](https://mlsanity.com/paper/1802.10503/) (2018)
- [Estimating Fingertip Forces, Torques, and Local Curvatures from Fingernail Images](https://mlsanity.com/paper/1909.05659/) (2019)
- [Prediction of Manipulation Actions](https://mlsanity.com/paper/1610.00759/) (2016)
- [Comparison of Constrained and Unconstrained Human Grasp Forces Using Fingernail Imaging and Visual Servoing](https://mlsanity.com/paper/2004.02311/) (2020)

## Built on by (most cited first, 3 of 3)

- [Force-Aware Interface via Electromyography for Natural VR/AR Interaction](https://mlsanity.com/paper/2210.01225/) (2022)
- [Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation](https://mlsanity.com/paper/2410.23629/) (2024)
- [EgoTac: In-the-wild Tactile Prediction from Egocentric Vision](https://mlsanity.com/paper/2608.15060/) (2026)

## Abstract (arXiv)

This paper has introduced a novel approach for the real-time estimation of 3D tactile forces exerted by human fingertips via vision only. The introduced approach is entirely monocular vision-based and does not require any physical force sensor. Therefore, it is scalable, non-intrusive, and easily fused with other perception systems such as body pose estimation, making it ideal for HRI applications where force sensing is necessary. The introduced approach consists of three main modules: finger tracking for detection and tracking of each individual finger, image alignment for preserving the spatial information in the images, and the force model for estimating the 3D forces from coloration patterns in the images. The model has been implemented experimentally, and the results have shown a maximum RMS error of 8.4% (for the entire range of force levels) along all three directions. The estimation accuracy is comparable to the offline models in the literature, such as EigneNail, while, this model is capable of performing force estimation at 30 frames per second.