mlsanityData to 7 Oct 2026

Human Hands as Probes for Interactive Object Understanding

arXiv 2112.09120 ↗First posted 16 Dec 2021cs.CV cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the EPIC-KITCHENS dataset, and successfully learn state-sensitive features, and object affordances (regions of interaction and afforded grasps), purely by observing hands in egocentric videos.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 50 of its 80 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

50 of 80 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2017Mask R-CNNcited by 7,259
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  10. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  12. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
38 more
  1. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  2. 2018SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networkscited by 445
  3. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  4. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  5. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  6. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  7. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  8. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  9. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  10. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  11. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  12. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  13. 2019Learning Individual Styles of Conversational Gesturecited by 190
  14. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  15. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  16. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  17. 2019Category-Level Articulated Object Pose Estimationcited by 158
  18. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  19. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  20. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  21. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  22. 2013Deep Learning for Detecting Robotic Graspscited by 114
  23. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  24. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  25. 2019You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactionscited by 82
  26. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  27. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  28. 2017SalGAN: Visual Saliency Prediction with Generative Adversarial Networkscited by 63
  29. 2018Binge Watching: Scaling Affordance Learning from Sitcomscited by 41
  30. 2016First-Person Activity Forecasting with Online Inverse Reinforcement Learningcited by 40
  31. 2020Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effectscited by 40
  32. 2019Learning Task-Oriented Grasping from Human Activity Datasetscited by 32
  33. 2019Analysis of the hands in egocentric vision: A surveycited by 25
  34. 2021The Functional Correspondence Problemcited by 20
  35. 2015You-Do, I-Learn: Unsupervised Multi-User egocentric Approach Towards Video-Based Guidancecited by 17
  36. 2021Shaping embodied agent behavior with activity-context priors from egocentric videocited by 17
  37. 2020Learning Object Manipulation Skills via Approximate State Estimation from Real Videoscited by 12
  38. 2020Hidden Footprints: Learning Contextual Walkability from 3D Human Trailscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 13; 2024: 18; 2025: 10; 2026: 7.

51 papers.

Sort:

By area:

  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  3. 2023Structured World Models from Human Videoscited by 144
  4. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  5. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  6. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  7. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  8. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  9. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  10. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  11. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  12. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
38 more
  1. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  2. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  3. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  4. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  5. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  6. 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 37
  7. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  8. 2023Understanding 3D Object Interaction from a Single Imagecited by 33
  9. 2023Pretrained Language Models as Visual Planners for Human Assistancecited by 28
  10. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  11. 2023Self-Supervised Category-Level Articulated Object Pose Estimation with Part-Level SE(3) Equivariancecited by 26
  12. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  13. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  14. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  15. 2023Multi-label affordance mapping from egocentric visioncited by 18
  16. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  17. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  18. 2024Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamicscited by 15
  19. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  20. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  21. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  22. 20252HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videoscited by 9
  23. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  24. 2025An Egocentric Vision-Language Model based Portable Real-time Smart Assistantcited by 7
  25. 2025OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfercited by 6
  26. 2024Self-Explainable Affordance Learning with Embodied Captioncited by 4
  27. 2025Ego-centric Predictive Model Conditioned on Hand Trajectoriescited by 4
  28. 2026ObjectForesight: Predicting Future 3D Object Trajectories from Human Videoscited by 4
  29. 2024One-shot Video Imitation via Parameterized Symbolic Abstraction Graphscited by 3
  30. 2025Affordance-Guided Diffusion Prior for 3D Hand Reconstructioncited by 2
  31. 2025Trajectory Conditioned Cross-embodiment Skill Transfercited by 2
  32. 2026Detecting Precise Hand Touch Moments in Egocentric Videocited by 2
  33. 2026What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?cited by 2
  34. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 2
  35. 2026Integrating Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 1
  36. 2026Flow as Flow: Modeling Robot Velocity Fields as Probability Velocity Fields for Flow-Based Object Manipulationcited by 1
  37. 2026MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Predictioncited by 1
  38. 2023InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wildcited by 0

50 most cited of 51 All 51 (JSON)