mlsanityData to 14 Oct 2026

OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction

arXiv 2203.15709 ↗First posted 29 Mar 2022cs.CVcited by 131

Not on the map. Why?

Abstract (arXiv)

Learning how humans manipulate objects requires machines to acquire knowledge from two perspectives: one for understanding object affordances and the other for learning human's interactions based on the affordances. Even though these two knowledge bases are crucial, we find that current databases lack a comprehensive awareness of them. In this work, we propose a multi-modal and rich-annotated knowledge repository, OakInk, for visual and cognitive understanding of hand-object interactions. We start to collect 1,800 common household objects and annotate their affordances to construct the first knowledge base: Oak. Given the affordance, we record rich human interactions with 100 selected objects in Oak. Finally, we transfer the interactions on the 100 recorded objects to their virtual counterparts through a novel method: Tink. The recorded and transferred hand-object interactions constitute the second knowledge base: Ink. As a result, OakInk contains 50,000 distinct affordance-aware and intent-oriented hand-object interactions. We benchmark OakInk on pose estimation and grasp generation tasks. Moreover, we propose two practical applications of OakInk: intent-based interaction generation and handover generation. Our datasets and source code are publicly available at https://github.com/lixiny/OakInk.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

43 of 65 references.

References · 43

  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  3. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  4. 2017Neural 3D Mesh Renderercited by 437
  5. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  6. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  7. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  8. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  9. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  10. 2017Integral Human Pose Regressioncited by 256
  11. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  12. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
31 more
  1. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
  2. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  3. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
  4. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
  5. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 182
  6. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  7. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  8. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 147
  9. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  10. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 141
  11. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  12. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  13. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
  14. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 128
  15. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
  16. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  17. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 114
  18. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  19. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  20. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 91
  21. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  22. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  23. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 58
  24. 2018Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervisioncited by 52
  25. 2017BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysiscited by 46
  26. 2019Learning Task-Oriented Grasping from Human Activity Datasetscited by 32
  27. 20173D Object Reconstruction from Hand-Object Interactionscited by 31
  28. 2018Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitationcited by 25
  29. 2021H2O: A Benchmark for Visual Human-human Object Handover Analysiscited by 16
  30. 2021HandTailor: Towards High-Precision Monocular 3D Hand Recoverycited by 15
  31. 2020Human Grasp Classification for Reactive Human-to-Robot Handoverscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 41; 2025: 31; 2026: 44.

131 papers.

Sort:

By area:

  1. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 88
  2. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  3. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  4. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 66
  5. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 62
  6. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 50
  7. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 49
  8. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  9. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  10. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  11. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  12. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Datacited by 37
38 more
  1. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 36
  2. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 35
  3. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 34
  4. 2024RealDex: Towards Human-like Grasping for Robotic Dexterous Handcited by 34
  5. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  6. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 33
  7. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 32
  8. 2024SemGrasp: Semantic Grasp Generation via Language Aligned Discretizationcited by 29
  9. 2024HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interactioncited by 29
  10. 2025A Survey on Human Interaction Motion Generationcited by 25
  11. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
  12. 2024ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusioncited by 21
  13. 2025Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomycited by 20
  14. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scalecited by 20
  15. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 19
  16. 2024ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Graspingcited by 18
  17. 2023ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimationcited by 16
  18. 2023SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstructioncited by 16
  19. 2024Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamicscited by 15
  20. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 15
  21. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 14
  22. 2023PressureVision++: Estimating Fingertip Pressure from Diverse RGB Imagescited by 12
  23. 20233D Reconstruction of Objects in Hands without Real World 3D Supervisioncited by 12
  24. 2023MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervisioncited by 12
  25. 2024Multi-GraspLLM: A Multimodal LLM for Multi-Hand Semantic Guided Grasp Generationcited by 12
  26. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  27. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  28. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 12
  29. 2023CHORD: Category-level Hand-held Object Reconstruction via Shape Deformationcited by 11
  30. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  31. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 11
  32. 2023POEM: Reconstructing Hand in a Point Embedded Multi-view Stereocited by 10
  33. 2023Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Imagescited by 10
  34. 2023Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videoscited by 10
  35. 2024POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View Worldcited by 10
  36. 2024NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Modelcited by 10
  37. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  38. 2025METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Modelcited by 10

50 most cited of 131 All 131 (JSON)