mlsanityData to 7 Oct 2026

Egocentric Hand-object Interaction Detection and Application

arXiv 2109.14734 ↗First posted 29 Sep 2021cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a method to detect the hand-object interaction from an egocentric perspective. In contrast to massive data-driven discriminator based method like citeShan20, we propose a novel workflow that utilises the cues of hand and object. Specifically, we train networks predicting hand pose, hand mask and in-hand object mask to jointly predict the hand-object interaction status. We compare our method with the most recent work from Shan et al. citeShan20 on selected images from EPIC-KITCHENS citedamen2018scaling dataset and achieve 89% accuracy on HOI (hand-object interaction) detection which is comparative to Shan's (92%). However, for real-time performance, with the same machine, our method can run over 30 FPS which is much efficient than Shan's (1~2 FPS). Furthermore, with our approach, we are able to segment script-less activities from where we extract the frames with the HOI status detection. We achieve 68.2% and 82.8% F1 score on GTEA citefathi2011learning and the UTGrasp citecai2015scalable dataset respectively which are all comparative to the SOTA methods.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 14 of its 27 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

14 of 27 references.

References · 14

  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  2. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  3. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  4. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  5. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  6. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  7. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  8. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  9. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  10. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  11. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  12. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
2 more
  1. 2020Two-hand Global 3D Pose Estimation Using Monocular RGBcited by 27
  2. 20143D Hand Pose Detection in Egocentric RGB-D Imagescited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 1; 2026: 3.

7 papers.

  1. 2023Exploiting Multimodal Synthetic Data for Egocentric Human-Object Interaction Detection in an Industrial Scenariocited by 14
  2. 2023ENIGMA-51: Towards a Fine-Grained Understanding of Human-Object Interactions in Industrial Scenarioscited by 11
  3. 2023Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?cited by 10
  4. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  5. 2026ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarioscited by 2
  6. 2026EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Groundingcited by 1
  7. 2026GlovEgo-HOI: Bridging the Synthetic-to-Real Gap for Industrial Egocentric Human-Object Interaction Detectioncited by 0