# AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose

arXiv: https://arxiv.org/abs/2309.08942 · HTML: https://mlsanity.com/paper/2309.08942/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 16 Sep 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [Hand-Object Contact Consistency Reasoning for Human Grasps Generation](https://mlsanity.com/paper/2104.03304/) (2021)
- [Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2012.09856/) (2020)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022)
- [OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction](https://mlsanity.com/paper/2203.15709/) (2022)
- [HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation](https://mlsanity.com/paper/2004.00060/) (2020)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [ContactGrasp: Functional Multi-finger Grasp Synthesis from Contact](https://mlsanity.com/paper/1904.03754/) (2019)
- [Learning Dexterous Grasping with Object-Centric Visual Affordances](https://mlsanity.com/paper/2009.01439/) (2020)
- [Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands](https://mlsanity.com/paper/2208.12250/) (2022)
- [BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis](https://mlsanity.com/paper/1704.02612/) (2017)
- [Binge Watching: Scaling Affordance Learning from Sitcoms](https://mlsanity.com/paper/1804.03080/) (2018)
- [Deep Differentiable Grasp Planner for High-DOF Grippers](https://mlsanity.com/paper/2002.01530/) (2020)
- [PartAfford: Part-level Affordance Discovery from 3D Objects](https://mlsanity.com/paper/2202.13519/) (2022)
- [Affordances in Robotic Tasks -- A Survey](https://mlsanity.com/paper/2004.07400/) (2020)
- [Contact2Grasp: 3D Grasp Synthesis via Hand-Object Contact Constraint](https://mlsanity.com/paper/2210.09245/) (2022)
- [Learning Social Affordance for Human-Robot Interaction](https://mlsanity.com/paper/1604.03692/) (2016)
- [Visual Affordance and Function Understanding: A Survey](https://mlsanity.com/paper/1807.06775/) (2018)

## Built on by (most cited first, 30 of 49)

- [TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding](https://mlsanity.com/paper/2401.08399/) (2024)
- [OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion](https://mlsanity.com/paper/2403.19417/) (2024)
- [ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions](https://mlsanity.com/paper/2401.10232/) (2024)
- [AffordanceLLM: Grounding Affordance from Vision Language Models](https://mlsanity.com/paper/2401.06341/) (2024)
- [HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances](https://mlsanity.com/paper/2403.01693/) (2024)
- [HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects](https://mlsanity.com/paper/2407.12371/) (2024)
- [GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities](https://mlsanity.com/paper/2412.04244/) (2024)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds](https://mlsanity.com/paper/2502.20041/) (2025)
- [SemGrasp: Semantic Grasp Generation via Language Aligned Discretization](https://mlsanity.com/paper/2404.03590/) (2024)
- [HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction](https://mlsanity.com/paper/2406.06843/) (2024)
- [UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation](https://mlsanity.com/paper/2506.09284/) (2025)
- [Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy](https://mlsanity.com/paper/2504.18829/) (2025)
- [EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision](https://mlsanity.com/paper/2409.02224/) (2024)
- [HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions](https://mlsanity.com/paper/2403.18575/) (2024)
- [MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulation](https://mlsanity.com/paper/2503.11081/) (2025)
- [ContactDexNet: Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-object Contact Semantic Mapping](https://mlsanity.com/paper/2404.08844/) (2024)
- [RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping](https://mlsanity.com/paper/2507.23734/) (2025)
- [Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?](https://mlsanity.com/paper/2312.02672/) (2023)
- [NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Model](https://mlsanity.com/paper/2407.12727/) (2024)
- [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025)
- [DyTact: Capturing Dynamic Contacts in Hand-Object Manipulation](https://mlsanity.com/paper/2506.03103/) (2025)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [Ins-HOI: Instance Aware Human-Object Interactions Recovery](https://mlsanity.com/paper/2312.09641/) (2023)
- [Text2Grasp: Grasp synthesis by text prompts of object grasping parts](https://mlsanity.com/paper/2404.15189/) (2024)
- [ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos](https://mlsanity.com/paper/2602.06226/) (2026)
- [G-DexGrasp: Generalizable Dexterous Grasping Synthesis Via Part-Aware Prior Retrieval and Prior-Assisted Generation](https://mlsanity.com/paper/2503.19457/) (2025)
- [Joint Diffusion for Universal Hand-Object Grasp Generation](https://mlsanity.com/paper/2409.04560/) (2024)
- [PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model](https://mlsanity.com/paper/2410.11564/) (2024)
- [Universal Dexterous Functional Grasping via Demonstration-Editing Reinforcement Learning](https://mlsanity.com/paper/2512.13380/) (2025)

## Abstract (arXiv)

How human interact with objects depends on the functional roles of the target objects, which introduces the problem of affordance-aware hand-object interaction. It requires a large number of human demonstrations for the learning and understanding of plausible and appropriate hand-object interactions. In this work, we present AffordPose, a large-scale dataset of hand-object interactions with affordance-driven hand pose. We first annotate the specific part-level affordance labels for each object, e.g. twist, pull, handle-grasp, etc, instead of the general intents such as use or handover, to indicate the purpose and guide the localization of the hand-object interactions. The fine-grained hand-object interactions reveal the influence of hand-centered affordances on the detailed arrangement of the hand poses, yet also exhibit a certain degree of diversity. We collect a total of 26.7K hand-object interactions, each including the 3D object shape, the part-level affordance label, and the manually adjusted hand poses. The comprehensive data analysis shows the common characteristics and diversity of hand-object interactions per affordance via the parameter statistics and contacting computation. We also conduct experiments on the tasks of hand-object affordance understanding and affordance-oriented hand-object interaction generation, to validate the effectiveness of our dataset in learning the fine-grained hand-object interactions. Project page: https://github.com/GentlesJan/AffordPose.