# Detecting Hands and Recognizing Physical Contact in the Wild

arXiv: https://arxiv.org/abs/2010.09676 · HTML: https://mlsanity.com/paper/2010.09676/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 30 of 38)

- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations](https://mlsanity.com/paper/2209.13064/) (2022)
- [ContactOpt: Optimizing Contact to Improve Grasps](https://mlsanity.com/paper/2104.07267/) (2021)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction](https://mlsanity.com/paper/2209.12354/) (2022)
- [On Self-Contact and Human Pose](https://mlsanity.com/paper/2104.03176/) (2021)
- [Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications](https://mlsanity.com/paper/2208.03826/) (2022)
- [DECO: Dense Estimation of 3D Human-Scene Contact In The Wild](https://mlsanity.com/paper/2309.15273/) (2023)
- [ContactGen: Generative Contact Modeling for Grasp Generation](https://mlsanity.com/paper/2310.03740/) (2023)
- [LEMON: Learning 3D Human-Object Interaction Relation from 2D Images](https://mlsanity.com/paper/2312.08963/) (2023)
- [HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances](https://mlsanity.com/paper/2403.01693/) (2024)
- [Detecting Human-Object Contact in Images](https://mlsanity.com/paper/2303.03373/) (2023)
- [EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views](https://mlsanity.com/paper/2405.13659/) (2024)
- [PressureVision: Estimating Hand Pressure from a Single RGB Image](https://mlsanity.com/paper/2203.10385/) (2022)
- [Hand-Object Interaction Image Generation](https://mlsanity.com/paper/2211.15663/) (2022)
- [FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation](https://mlsanity.com/paper/2402.00293/) (2024)
- [HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision](https://mlsanity.com/paper/2404.09933/) (2024)
- [HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild](https://mlsanity.com/paper/2404.13819/) (2024)
- [SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognition](https://mlsanity.com/paper/2204.04796/) (2022)
- [Stability-driven Contact Reconstruction From Monocular Color Images](https://mlsanity.com/paper/2205.00848/) (2022)
- [Target-Aware Video Diffusion Models](https://mlsanity.com/paper/2503.18950/) (2025)
- [Precision-Enhanced Human-Object Contact Detection via Depth-Aware Perspective Interaction and Object Texture Restoration](https://mlsanity.com/paper/2412.09920/) (2024)
- [VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification](https://mlsanity.com/paper/2512.09646/) (2025)
- [FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understanding](https://mlsanity.com/paper/2603.15847/) (2026)
- [HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2412.01508/) (2024)
- [Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction](https://mlsanity.com/paper/2110.10174/) (2021)
- [BPJDet: Extended Object Representation for Generic Body-Part Joint Detection](https://mlsanity.com/paper/2304.10765/) (2023)
- [Detecting Precise Hand Touch Moments in Egocentric Video](https://mlsanity.com/paper/2604.12343/) (2026)
- [Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video](https://mlsanity.com/paper/2308.04074/) (2023)

## Abstract (arXiv)

We investigate a new problem of detecting hands and recognizing their physical contact state in unconstrained conditions. This is a challenging inference task given the need to reason beyond the local appearance of hands. The lack of training annotations indicating which object or parts of an object the hand is in contact with further complicates the task. We propose a novel convolutional network based on Mask-RCNN that can jointly learn to localize hands and predict their physical contact to address this problem. The network uses outputs from another object detector to obtain locations of objects present in the scene. It uses these outputs and hand locations to recognize the hand's contact state using two attention mechanisms. The first attention mechanism is based on the hand and a region's affinity, enclosing the hand and the object, and densely pools features from this region to the hand region. The second attention module adaptively selects salient features from this plausible region of contact. To develop and evaluate our method's performance, we introduce a large-scale dataset called ContactHands, containing unconstrained images annotated with hand locations and contact states. The proposed network, including the parameters of attention modules, is end-to-end trainable. This network achieves approximately 7% relative improvement over a baseline network that was built on the vanilla Mask-RCNN architecture and trained for recognizing hand contact states.