# Analysis of Hand Segmentation in the Wild

arXiv: https://arxiv.org/abs/1803.03317 · HTML: https://mlsanity.com/paper/1803.03317/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2018.

## Built on by (most cited first, 18 of 18)

- [Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications](https://mlsanity.com/paper/2208.03826/) (2022)
- [Forecasting Action through Contact Representations from First Person Video](https://mlsanity.com/paper/2102.00649/) (2021)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [Domain Adaptive Hand Keypoint and Pixel Localization in the Wild](https://mlsanity.com/paper/2203.08344/) (2022)
- [Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets](https://mlsanity.com/paper/2205.06218/) (2022)
- [Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition](https://mlsanity.com/paper/2110.15639/) (2021)
- [TriHorn-Net: A Model for Accurate Depth-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2206.07117/) (2022)
- [Vision as Unified Multimodal Generation](https://mlsanity.com/paper/2607.06560/) (2026)
- [Foreground-Aware Stylization and Consensus Pseudo-Labeling for Domain Adaptation of First-Person Hand Segmentation](https://mlsanity.com/paper/2107.02718/) (2021)
- [Ego2HandsPose: A Dataset for Egocentric Two-hand 3D Global Pose Estimation](https://mlsanity.com/paper/2206.04927/) (2022)
- [DACov: A Deeper Analysis of Data Augmentation on the Computed Tomography Segmentation Problem](https://mlsanity.com/paper/2303.05912/) (2023)
- [Benchmarking 2D Egocentric Hand Pose Datasets](https://mlsanity.com/paper/2409.07337/) (2024)
- [EgoSurgery-HTS: A Dataset for Egocentric Hand-Tool Segmentation in Open Surgery Videos](https://mlsanity.com/paper/2503.18755/) (2025)
- [The Collection of a Human Robot Collaboration Dataset for Cooperative Assembly in Glovebox Environments](https://mlsanity.com/paper/2407.14649/) (2024)
- [Challenges of the Creation of a Dataset for Vision Based Human Hand Action Recognition in Industrial Assembly](https://mlsanity.com/paper/2303.03716/) (2023)
- [Improving and Evaluating Hand-Object Interaction Detection](https://mlsanity.com/paper/2606.17384/) (2026)
- [Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots](https://mlsanity.com/paper/2102.04750/) (2021)
- [Testing Human-Hand Segmentation on In-Distribution and Out-of-Distribution Data in Human-Robot Interactions Using a Deep Ensemble Model](https://mlsanity.com/paper/2501.07713/) (2025)

## Abstract (arXiv)

A large number of works in egocentric vision have concentrated on action and object recognition. Detection and segmentation of hands in first-person videos, however, has less been explored. For many applications in this domain, it is necessary to accurately segment not only hands of the camera wearer but also the hands of others with whom he is interacting. Here, we take an in-depth look at the hand segmentation problem. In the quest for robust hand segmentation methods, we evaluated the performance of the state of the art semantic segmentation methods, off the shelf and fine-tuned, on existing datasets. We fine-tune RefineNet, a leading semantic segmentation method, for hand segmentation and find that it does much better than the best contenders. Existing hand segmentation datasets are collected in the laboratory settings. To overcome this limitation, we contribute by collecting two new datasets: a) EgoYouTubeHands including egocentric videos containing hands in the wild, and b) HandOverFace to analyze the performance of our models in presence of similar appearance occlusions. We further explore whether conditional random fields can help refine generated hand segmentations. To demonstrate the benefit of accurate hand maps, we train a CNN for hand-based activity recognition and achieve higher accuracy when a CNN was trained using hand maps produced by the fine-tuned RefineNet. Finally, we annotate a subset of the EgoHands dataset for fine-grained action recognition and show that an accuracy of 58.6% can be achieved by just looking at a single hand pose which is much better than the chance level (12.5%).