# Fine-grained Affordance Annotation for Egocentric Hand-Object Interaction Videos

arXiv: https://arxiv.org/abs/2302.03292 · HTML: https://mlsanity.com/paper/2302.03292/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 12 of 12)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018)
- [EGO-TOPO: Environment Affordances from Egocentric Video](https://mlsanity.com/paper/2001.04583/) (2020)
- [Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition](https://mlsanity.com/paper/1803.09125/) (2018)
- [Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions](https://mlsanity.com/paper/1901.01874/) (2019)
- [Deep Affordance-grounded Sensorimotor Object Recognition](https://mlsanity.com/paper/1704.02787/) (2017)

## Built on by (most cited first, 15 of 15)

- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [Learning Precise Affordances from Egocentric Videos for Robotic Manipulation](https://mlsanity.com/paper/2408.10123/) (2024)
- [EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views](https://mlsanity.com/paper/2405.13659/) (2024)
- [EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs](https://mlsanity.com/paper/2507.18342/) (2025)
- [Multi-label affordance mapping from egocentric vision](https://mlsanity.com/paper/2309.02120/) (2023)
- [FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotation](https://mlsanity.com/paper/2402.00293/) (2024)
- [Diagnosing Human-object Interaction Detectors](https://mlsanity.com/paper/2308.08529/) (2023)
- [Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos](https://mlsanity.com/paper/2311.16444/) (2023)
- [Text-driven Affordance Learning from Egocentric Vision](https://mlsanity.com/paper/2404.02523/) (2024)
- [Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors](https://mlsanity.com/paper/2505.24103/) (2025)
- [EMAG: Ego-motion Aware and Generalizable 2D Hand Forecasting from Egocentric Videos](https://mlsanity.com/paper/2405.20030/) (2024)
- [Egocentric zone-aware action recognition across environments](https://mlsanity.com/paper/2409.14205/) (2024)
- [ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction](https://mlsanity.com/paper/2504.01472/) (2025)
- [Proposal-based Temporal Action Localization with Point-level Supervision](https://mlsanity.com/paper/2310.05511/) (2023)
- [Affordance-Guided Diffusion Prior for 3D Hand Reconstruction](https://mlsanity.com/paper/2510.00506/) (2025)

## Abstract (arXiv)

Object affordance is an important concept in hand-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning. However, the definition of affordance in existing datasets often: 1) mix up affordance with object functionality; 2) confuse affordance with goal-related action; and 3) ignore human motor capacity. This paper proposes an efficient annotation scheme to address these issues by combining goal-irrelevant motor actions and grasp types as affordance labels and introducing the concept of mechanical action to represent the action possibilities between two objects. We provide new annotations by applying this scheme to the EPIC-KITCHENS dataset and test our annotation with tasks such as affordance recognition, hand-object interaction hotspots prediction, and cross-domain evaluation of affordance. The results show that models trained with our annotation can distinguish affordance from other concepts, predict fine-grained interaction possibilities on objects, and generalize through different domains.