mlsanityData to 14 Oct 2026

Multi-label affordance mapping from egocentric vision

arXiv 2309.02120 ↗First posted 5 Sep 2023cs.CV cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Accurate affordance detection and segmentation with pixel precision is an important piece in many complex systems based on interactions, such as robots and assitive devices. We present a new approach to affordance perception which enables accurate multi-label segmentation. Our approach can be used to automatically extract grounded affordances from first person videos of interactions using a 3D map of the environment providing pixel level precision for the affordance location. We use this method to build the largest and most complete dataset on affordances based on the EPIC-Kitchen dataset, EPIC-Aff, which provides interaction-grounded, multi-label, metric and spatial affordance annotations. Then, we propose a new approach to affordance segmentation based on multi-label detection which enables multiple affordances to co-exists in the same space, for example if they are associated with the same object. We present several strategies of multi-label detection using several segmentation architectures. The experimental results highlight the importance of the multi-label detection. Finally, we show how our metric representation can be exploited for build a map of interaction hotspots in spatial action-centric zones and use that representation to perform a task-oriented navigation.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 52 of its 80 references that are in the map. First posted on arXiv 5 Sep 2023.

Builds on

52 of 80 references.

References · 52

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2017Mask R-CNNcited by 7,277
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  6. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  7. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  8. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  9. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 988
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  11. 2017Focal Loss for Dense Object Detectioncited by 752
  12. 2019Panoptic Feature Pyramid Networkscited by 645
38 more
  1. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  2. 2020Asymmetric Loss For Multi-Label Classificationcited by 381
  3. 2019Multi-Label Image Recognition with Graph Convolutional Networkscited by 322
  4. 2021Anticipative Video Transformercited by 200
  5. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  6. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  7. 2018Grounded Human-Object Interaction Hotspots from Videocited by 129
  8. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 128
  9. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 128
  10. 2020General Multi-label Image Classification with Transformerscited by 127
  11. 2020Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasetscited by 122
  12. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  13. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  14. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  15. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
  16. 2015Semantic Amodal Segmentationcited by 103
  17. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  18. 2020Attention-Driven Dynamic Graph Convolutional Network for Multi-Label Image Recognitioncited by 80
  19. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  20. 2017Learning to Act Properly: Predicting and Explaining Affordances from Imagescited by 74
  21. 2015Efficient piecewise training of deep structured models for semantic segmentationcited by 71
  22. 2020Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Viewscited by 58
  23. 2019Next-Active-Object prediction from Egocentric Videoscited by 55
  24. 2021Human Hands as Probes for Interactive Object Understandingcited by 52
  25. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  26. 2016First-Person Activity Forecasting with Online Inverse Reinforcement Learningcited by 40
  27. 2021One-Shot Object Affordance Detection in the Wildcited by 40
  28. 2022Amodal Panoptic Segmentationcited by 38
  29. 2018Chest X-rays Classification: A Multi-Label and Fine-Grained Problemcited by 31
  30. 2017Joint Discovery of Object States and Manipulation Actionscited by 28
  31. 2019Learning Grasp Affordance Reasoning through Semantic Relationscited by 23
  32. 2021MlTr: Multi-label Classification with Transformercited by 21
  33. 2019Generative Hybrid Representations for Activity Forecasting with No-Regret Learningcited by 18
  34. 2015You-Do, I-Learn: Unsupervised Multi-User egocentric Approach Towards Video-Based Guidancecited by 17
  35. 2023Fine-grained Affordance Annotation for Egocentric Hand-Object Interaction Videoscited by 15
  36. 2023Bayesian Deep Learning for Affordance Segmentation in imagescited by 13
  37. 2016Learning Action Maps of Large Environments via First-Person Visioncited by 11
  38. 2022RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentationcited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  2. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  3. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  4. 2024Learning 2D Invariant Affordance Knowledge for 3D Affordance Groundingcited by 22
  5. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  6. 2024Text-driven Affordance Learning from Egocentric Visioncited by 9
  7. 2024FIction: 4D Future Interaction Prediction from Videocited by 9
  8. 2025O-MaMa: Learning Object Mask Matching between Egocentric and Exocentric Viewscited by 7
  9. 2024Egocentric zone-aware action recognition across environmentscited by 6
  10. 2025Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scalecited by 6
  11. 2025Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalitiescited by 1
  12. 2025Visual Affordance Prediction: Survey and Reproducibilitycited by 1
6 more
  1. 2025Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mappingcited by 1
  2. 2026Integrating Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 1
  3. 2025DIV-FF: Dynamic Image-Video Feature Fields For Environment Understanding in Egocentric Videoscited by 0
  4. 2026Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformerscited by 0
  5. 2026Token-Based Affordance Grounding with Large Vision-Language Modelscited by 0
  6. 2026Reproducible Multimodal Affordance Predictioncited by 0