# Affordance Grounding from Demonstration Video to Target Image

arXiv: https://arxiv.org/abs/2303.14644 · HTML: https://mlsanity.com/paper/2303.14644/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 62 references that are in the map.
- First posted on arXiv 26 Mar 2023.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)
- [Actor and Observer: Joint Modeling of First and Third-Person Videos](https://mlsanity.com/paper/1804.09627/) (2018)

## Built on by (most cited first, 30 of 34)

- [AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn](https://mlsanity.com/paper/2306.08640/) (2023)
- [DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)](https://mlsanity.com/paper/2401.08392/) (2024)
- [AffordanceLLM: Grounding Affordance from Vision Language Models](https://mlsanity.com/paper/2401.06341/) (2024)
- [One-Shot Open Affordance Learning with Foundation Models](https://mlsanity.com/paper/2311.17776/) (2023)
- [Learning Manipulation by Predicting Interaction](https://mlsanity.com/paper/2406.00439/) (2024)
- [3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds](https://mlsanity.com/paper/2502.20041/) (2025)
- [PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2601.07060/) (2026)
- [GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency](https://mlsanity.com/paper/2412.09511/) (2024)
- [WorldAfford: Affordance Grounding based on Natural Language Instructions](https://mlsanity.com/paper/2405.12461/) (2024)
- [INTRA: Interaction Relationship-aware Weakly Supervised Affordance Grounding](https://mlsanity.com/paper/2409.06210/) (2024)
- [RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping](https://mlsanity.com/paper/2507.23734/) (2025)
- [Text-driven Affordance Learning from Egocentric Vision](https://mlsanity.com/paper/2404.02523/) (2024)
- [Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors](https://mlsanity.com/paper/2505.24103/) (2025)
- [EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observations](https://mlsanity.com/paper/2506.17896/) (2025)
- [Egocentric and Exocentric Methods: A Short Survey](https://mlsanity.com/paper/2410.20621/) (2024)
- [AnchorDP3: 3D Affordance Guided Sparse Diffusion Policy for Robotic Manipulation](https://mlsanity.com/paper/2506.19269/) (2025)
- [Closed-Loop Transfer for Weakly-supervised Affordance Grounding](https://mlsanity.com/paper/2510.17384/) (2025)
- [First Place Solution to the CVPR'2023 AQTC Challenge: A Function-Interaction Centric Approach with Spatiotemporal Visual-Language Alignment](https://mlsanity.com/paper/2306.13380/) (2023)
- [Interpretable Affordance Detection on 3D Point Clouds with Probabilistic Prototypes](https://mlsanity.com/paper/2504.18355/) (2025)
- [Visual-Geometric Collaborative Guidance for Affordance Learning](https://mlsanity.com/paper/2410.11363/) (2024)
- [VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Model](https://mlsanity.com/paper/2602.09638/) (2026)
- [Populate-A-Scene: Affordance-Aware Human Video Generation](https://mlsanity.com/paper/2507.00334/) (2025)
- [H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows](https://mlsanity.com/paper/2510.21769/) (2025)
- [VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videos](https://mlsanity.com/paper/2602.20608/) (2026)
- [CompassAD: Intent-Driven 3D Affordance Grounding in Functionally Competing Objects](https://mlsanity.com/paper/2604.02060/) (2026)
- [Leverage Task Context for Object Affordance Ranking](https://mlsanity.com/paper/2411.16082/) (2024)
- [Capturing Fine-Grained Alignments Improves 3D Affordance Detection](https://mlsanity.com/paper/2506.19312/) (2025)
- [TReF-6: Inferring Task-Relevant Frames from a Single Demonstration for One-Shot Skill Generalization](https://mlsanity.com/paper/2509.00310/) (2025)
- [Learning Visual Affordance from Audio](https://mlsanity.com/paper/2512.02005/) (2025)
- [Probing and Bridging Geometry-Interaction Cues for Affordance Reasoning in Vision Foundation Models](https://mlsanity.com/paper/2602.20501/) (2026)

## Abstract (arXiv)

Humans excel at learning from expert demonstrations and solving their own problems. To equip intelligent robots and assistants, such as AR glasses, with this ability, it is essential to ground human hand interactions (i.e., affordances) from demonstration videos and apply them to a target image like a user's AR glass view. The video-to-image affordance grounding task is challenging due to (1) the need to predict fine-grained affordances, and (2) the limited training data, which inadequately covers video-image discrepancies and negatively impacts grounding. To tackle them, we propose Affordance Transformer (Afformer), which has a fine-grained transformer-based decoder that gradually refines affordance grounding. Moreover, we introduce Mask Affordance Hand (MaskAHand), a self-supervised pre-training technique for synthesizing video-image data and simulating context changes, enhancing affordance grounding across video-image discrepancies. Afformer with MaskAHand pre-training achieves state-of-the-art performance on multiple benchmarks, including a substantial 37% improvement on the OPRA dataset. Code is made available at https://github.com/showlab/afformer.