mlsanityData to 7 Oct 2026

H2O: Two Hands Manipulating Objects for First Person Interaction Recognition

arXiv 2104.11181 ↗First posted 22 Apr 2021cs.CVcited by 251

Not on the map. Why?

Abstract (arXiv)

We present a comprehensive framework for egocentric interaction recognition using markerless 3D annotations of two hands manipulating objects. To this end, we propose a method to create a unified dataset for egocentric 3D interaction recognition. Our method produces annotations of the 3D pose of two hands and the 6D pose of the manipulated objects, along with their interaction labels for each frame. Our dataset, called H2O (2 Hands and Objects), provides synchronized multi-view RGB-D images, interaction labels, object classes, ground-truth 3D poses for left & right hands, 6D object poses, ground-truth camera poses, object meshes and scene point clouds. To the best of our knowledge, this is the first benchmark that enables the study of first-person actions with the use of the pose of both left and right hands manipulating objects and presents an unprecedented level of detail for egocentric 3D interaction recognition. We further propose the method to predict interaction classes by estimating the 3D pose of two hands and the 6D pose of the manipulated objects, jointly from RGB images. Our method models both inter- and intra-dependencies between both hands and objects by learning the topology of a graph convolutional network that predicts interactions. We show that our method facilitated by this dataset establishes a strong baseline for joint hand-object pose estimation and achieves state-of-the-art accuracy for first person interaction recognition.

Key facts. Cited by 251 papers in the map (data through 7 Oct 2026). Builds on 64 of its 103 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

64 of 103 references.

References · 64

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Mask R-CNNcited by 7,259
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  9. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  10. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
38 more
  1. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  2. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  3. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  4. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  5. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  7. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  8. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  9. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  10. 2018PVNet: Pixel-wise Voting Network for 6DoF Pose Estimationcited by 385
  11. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  12. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  13. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  14. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  15. 2017Temporal Relational Reasoning in Videoscited by 324
  16. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  17. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  18. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  19. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  20. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  21. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  22. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  23. 2019Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognitioncited by 228
  24. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  25. 2017Real-Time Seamless Single Shot 6D Object Pose Predictioncited by 210
  26. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  27. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  28. 2016Panoptic Studio: A Massively Multiview System for Social Interaction Capturecited by 173
  29. 2019The VIA Annotation Software for Images, Audio and Videocited by 166
  30. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  31. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  32. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  33. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  34. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  35. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  36. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 102
  37. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  38. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 22; 2023: 36; 2024: 48; 2025: 65; 2026: 72.

251 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  3. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  4. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  5. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  6. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  7. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  8. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  9. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  10. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  11. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  12. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
38 more
  1. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  2. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  3. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  4. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  5. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  6. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interactioncited by 84
  7. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  8. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  9. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  10. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  11. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  12. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 69
  13. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  14. 2022LISA: Learning Implicit Shape and Appearance of Handscited by 63
  15. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 60
  16. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  17. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 57
  18. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  19. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  20. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  21. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  22. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  23. 2024Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Cameracited by 50
  24. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 48
  25. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  26. 2024SkateFormer: Skeletal-Temporal Transformer for Human Action Recognitioncited by 46
  27. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  28. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  29. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  30. 2023Bringing Inputs to Shared Domains for 3D Interacting Hands Recovery in the Wildcited by 36
  31. 2023Object pop-up: Can we infer 3D objects and their poses from human interactions alone?cited by 36
  32. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 36
  33. 2022Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videoscited by 35
  34. 2023A Dataset of Relighted 3D Interacting Handscited by 35
  35. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  36. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
  37. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  38. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33

50 most cited of 251 All 251 (JSON)