mlsanityData to 7 Oct 2026

H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

arXiv 1904.05349 ↗First posted 10 Apr 2019cs.CVcited by 139

Not on the map. Why?

Abstract (arXiv)

We present a unified framework for understanding 3D hand and object interactions in raw image sequences from egocentric RGB cameras. Given a single RGB image, our model jointly estimates the 3D hand and object poses, models their interactions, and recognizes the object and action classes with a single feed-forward pass through a neural network. We propose a single architecture that does not rely on external detection algorithms but rather is trained end-to-end on single images. We further merge and propagate information in the temporal domain to infer interactions between hand and object trajectories and recognize actions. The complete model takes as input a sequence of frames and outputs per-frame 3D hand and object pose predictions along with the estimates of object and action categories for the entire sequence. We demonstrate state-of-the-art performance of our algorithm even in comparison to the approaches that work on depth data and ground-truth annotations.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 24; 2023: 34; 2024: 23; 2025: 15; 2026: 11.

139 papers.

Sort:

By area:

  1. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  2. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  3. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  4. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  5. 2021Towards Long-Form Video Understandingcited by 146
  6. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  7. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  8. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  9. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  10. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  11. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  12. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
38 more
  1. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  2. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  3. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  4. 2023An Outlook into the Future of Egocentric Visioncited by 84
  5. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  6. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  7. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  8. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  9. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 59
  10. 2022AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstructioncited by 58
  11. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 57
  12. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  13. 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
  14. 2023CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesiscited by 54
  15. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  16. 2023ContactGen: Generative Contact Modeling for Grasp Generationcited by 52
  17. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  18. 2021A Skeleton-Driven Neural Occupancy Representation for Articulated Handscited by 50
  19. 2021Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspectivecited by 49
  20. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 49
  21. 2021Forecasting Action through Contact Representations from First Person Videocited by 46
  22. 2021Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstructioncited by 44
  23. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  24. 2021Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentationcited by 40
  25. 2021Trear: Transformer-based RGB-D Egocentric Action Recognitioncited by 39
  26. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  27. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  28. 2021Predicting the Future from First Person (Egocentric) Vision: A Surveycited by 36
  29. 2022Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videoscited by 35
  30. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  31. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  32. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 33
  33. 2022HARP: Personalized Hand Reconstruction from a Monocular RGB Videocited by 28
  34. 2022Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motioncited by 27
  35. 2021RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Videocited by 23
  36. 2023Interactive Spatiotemporal Token Attention Network for Skeleton-based General Interactive Action Recognitioncited by 23
  37. 2023HMP: Hand Motion Priors for Pose and Shape Estimation from Videocited by 23
  38. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23

50 most cited of 139 All 139 (JSON)