mlsanityData to 7 Oct 2026

Understanding Human Hands in Contact at Internet Scale

arXiv 2006.06669 ↗First posted 11 Jun 2020cs.CVcited by 281

Not on the map. Why?

Abstract (arXiv)

Hands are the central means by which humans manipulate their world and being able to reliably extract hand state information from Internet videos of humans engaged in their hands has the potential to pave the way to systems that can learn from petabytes of video data. This paper proposes steps towards this by inferring a rich representation of hands engaged in interaction method that includes: hand location, side, contact state, and a box around the object in contact. To support this effort, we gather a large-scale dataset of hands in contact with objects consisting of 131 days of footage as well as a 100K annotated hand-contact video frame dataset. The learned model on this dataset can serve as a foundation for hand-contact understanding in videos. We quantitatively evaluate it both on its own and in service of predicting and learning from 3D meshes of human hands.

Key facts. Cited by 281 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 44; 2023: 62; 2024: 65; 2025: 48; 2026: 36.

281 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  3. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  4. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  5. 2023Reconstructing Hands in 3D with Transformerscited by 298
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  7. 2022Masked Visual Pre-training for Motor Controlcited by 246
  8. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  9. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  11. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  12. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
38 more
  1. 2023Language-Driven Representation Learning for Roboticscited by 163
  2. 2022Human-to-Robot Imitation in the Wildcited by 157
  3. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  4. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  5. 2023Structured World Models from Human Videoscited by 144
  6. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  7. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  8. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  9. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 117
  10. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  11. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  12. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  13. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  14. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  15. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  16. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  17. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
  18. 2023An Outlook into the Future of Egocentric Visioncited by 84
  19. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  20. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  21. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  22. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  23. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  24. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  25. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  26. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  27. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 57
  28. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  29. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  30. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  31. 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
  32. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  33. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  34. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  35. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  36. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  37. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  38. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49

50 most cited of 281 All 281 (JSON)