mlsanityData to 14 Oct 2026

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications

arXiv 2208.03826 ↗First posted 7 Aug 2022cs.CVcited by 78

Not on the map. Why?

Abstract (arXiv)

Egocentric videos offer fine-grained information for high-fidelity modeling of human behaviors. Hands and interacting objects are one crucial aspect of understanding a viewer's behaviors and intentions. We provide a labeled dataset consisting of 11,243 egocentric images with per-pixel segmentation labels of hands and objects being interacted with during a diverse array of daily activities. Our dataset is the first to label detailed hand-object contact boundaries. We introduce a context-aware compositional data augmentation technique to adapt to out-of-distribution YouTube egocentric video. We show that our robust hand-object segmentation model and dataset can serve as a foundational tool to boost or enable several downstream vision applications, including hand state classification, video activity recognition, 3D mesh reconstruction of hand-object interactions, and video inpainting of hand-object foregrounds in egocentric videos. Dataset and code are available at: https://github.com/owenzlz/EgoHOS

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 57 of its 85 references that are in the map. First posted on arXiv 7 Aug 2022.

Builds on

57 of 85 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  11. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  12. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
38 more
  1. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  2. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 465
  3. 2019PointRend: Image Segmentation as Renderingcited by 440
  4. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  5. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  6. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  7. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  8. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  9. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
  10. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  11. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  12. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  13. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  14. 2020BoxInst: High-Performance Instance Segmentation with Box Annotationscited by 151
  15. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  16. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  17. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  18. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
  19. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  20. 2018Grounded Human-Object Interaction Hotspots from Videocited by 129
  21. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  22. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  23. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  24. 2020Flow-edge Guided Video Completioncited by 104
  25. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  26. 2019InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pastingcited by 95
  27. 2021On Self-Contact and Human Posecited by 89
  28. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 89
  29. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  30. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 85
  31. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 73
  32. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  33. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 58
  34. 2021Human Hands as Probes for Interactive Object Understandingcited by 52
  35. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  36. 2018Disentangling Latent Hands for Image Synthesis and Pose Estimationcited by 44
  37. 2020Detecting Hands and Recognizing Physical Contact in the Wildcited by 38
  38. 2019Single Image 3D Hand Reconstruction with Mesh Convolutionscited by 32

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 23; 2025: 23; 2026: 20.

78 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,234
  2. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  3. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 129
  4. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
  5. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  6. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  7. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 58
  8. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
  9. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 48
  10. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  11. 2023Detecting Human-Object Contact in Imagescited by 41
  12. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 39
38 more
  1. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  2. 2023What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentationcited by 24
  3. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  4. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 23
  5. 2022Grounded Affordance from Exocentric Viewcited by 22
  6. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  7. 2024Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videoscited by 21
  8. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  9. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  10. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  11. 2023Multi-label affordance mapping from egocentric visioncited by 18
  12. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 18
  13. 2024ZIM: Zero-Shot Image Matting for Anythingcited by 16
  14. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  15. 2025SAM2 for Image and Video Segmentation: A Comprehensive Surveycited by 13
  16. 2024FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotationcited by 12
  17. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  18. 2024ASAM: Boosting Segment Anything Model with Adversarial Tuningcited by 12
  19. 2023ENIGMA-51: Towards a Fine-Grained Understanding of Human-Object Interactions in Industrial Scenarioscited by 11
  20. 2024Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?cited by 11
  21. 2023Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?cited by 10
  22. 2024HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wildcited by 9
  23. 20252HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videoscited by 9
  24. 2024ActionVOS: Actions as Prompts for Video Object Segmentationcited by 8
  25. 2024CaRe-Ego: Contact-aware Relationship Modeling for Egocentric Interactive Hand-object Segmentationcited by 7
  26. 2026EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videoscited by 7
  27. 2024Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentationcited by 6
  28. 2025EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretrainingcited by 5
  29. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  30. 2025Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentationcited by 5
  31. 2026ObjectForesight: Predicting Future 3D Object Trajectories from Human Videoscited by 5
  32. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 5
  33. 2025EgoSplat: Open-Vocabulary Egocentric Scene Understanding with Language Embedded 3D Gaussian Splattingcited by 4
  34. 2026Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsingcited by 4
  35. 2025EgoSurgery-HTS: A Dataset for Egocentric Hand-Tool Segmentation in Open Surgery Videoscited by 3
  36. 2025SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectoriescited by 3
  37. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  38. 2025MultiModal Action Conditioned Video Generationcited by 3

50 most cited of 78 All 78 (JSON)