mlsanityData to 7 Oct 2026

Text-driven object affordance for guiding grasp-type recognition in multimodal robot teaching

arXiv 2103.00268 ↗First posted 27 Feb 2021cs.RO cs.CV cs.HCcited by 1

Not on the map. Why?

Abstract (arXiv)

This study investigates how text-driven object affordance, which provides prior knowledge about grasp types for each object, affects image-based grasp-type recognition in robot teaching. The researchers created labeled datasets of first-person hand images to examine the impact of object affordance on recognition performance. They evaluated scenarios with real and illusory objects, considering mixed reality teaching conditions where visual object information may be limited. The results demonstrate that object affordance improves image-based recognition by filtering out unlikely grasp types and emphasizing likely ones. The effectiveness of object affordance was more pronounced when there was a stronger bias towards specific grasp types for each object. These findings highlight the significance of object affordance in multimodal robot teaching, regardless of whether real objects are present in the images. Sample code is available on https://github.com/microsoft/arr-grasp-type-recognition.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 60 references that are in the map. First posted on arXiv 27 Feb 2021.

Builds on

18 of 60 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  3. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  4. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  5. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  6. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  7. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  8. 2013Deep Learning for Detecting Robotic Graspscited by 114
  9. 2013Data-Driven Grasp Synthesis - A Surveycited by 104
  10. 2014Real-Time Grasp Detection Using Convolutional Neural Networkscited by 75
  11. 2018Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervisioncited by 50
  12. 2019Learning Task-Oriented Grasping from Human Activity Datasetscited by 32
6 more
  1. 2019Learning to Estimate Pose and Shape of Hand-Held Objects from RGB Imagescited by 10
  2. 2020A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operationscited by 7
  3. 2018Understanding hand-object manipulation by modeling the contextual relationship between actions, grasp types and object attributescited by 4
  4. 2020Verbal Focus-of-Attention System for Learning-from-Observationcited by 3
  5. 2020Task-oriented Motion Mapping on Robots of Various Configuration using Body Role Divisioncited by 3
  6. 2020Grasp-type Recognition Leveraging Object Affordancecited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76