mlsanityData to 14 Oct 2026

Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions

arXiv 2505.05517 ↗First posted 7 May 2025cs.CV cs.LG cs.ROcited by 11

Not on the map. Why?

Abstract (arXiv)

Functional grasping is essential for enabling dexterous multi-finger robot hands to manipulate objects effectively. Prior work largely focuses on power grasps, which only involve holding an object, or relies on in-domain demonstrations for specific objects. We propose leveraging human grasp information extracted from web images, which capture natural and functional hand-object interactions (HOI). Using a pretrained 3D reconstruction model, we recover 3D human HOI meshes from RGB images. To train on these noisy HOI data, we propose to use: (1) an interaction-centric model to learn the functional interaction pattern between hand and object, and (2) geometry-based filtering to remove the infeasible grasps and physical simulation to retain grasps who can resist disturbance. In IssacGym simulation, our model trained on reconstructed HOI grasps achieves a 75.8% success rate on objects from the web dataset and generalizes to unseen objects, outperforming baseline methods in both grasp success and functional quality. In real-world experiments with the LEAP hand and Inspire hand, it attains a 77.5% success rate across 12 objects, including challenging ones such as a syringe, spray bottle, knife, and tongs. Project website is at: https://web2grasp.github.io/.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 7 May 2025.

Builds on

48 of 62 references.

References · 48

  1. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  2. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  6. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  7. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  8. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
  9. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  10. 2023Reconstructing Hands in 3D with Transformerscited by 306
  11. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  12. 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 208
36 more
  1. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  2. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 152
  3. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 147
  4. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
  5. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
  6. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 137
  7. 2017Learning Efficient Point Cloud Generation for Dense 3D Object Reconstructioncited by 133
  8. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 131
  9. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  10. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  11. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 107
  12. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  13. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  14. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  15. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  16. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  17. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  18. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 79
  19. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
  20. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  21. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 73
  22. 2021Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimatorcited by 71
  23. 2022GenDexGrasp: Generalizable Dexterous Graspingcited by 56
  24. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
  25. 2023Creative Robot Tool Use with Large Language Modelscited by 43
  26. 2024D(R,O) Grasp: A Unified Representation of Robot and Object Interaction for Cross-Embodiment Dexterous Graspingcited by 42
  27. 2023Dexterous Functional Graspingcited by 39
  28. 2024Hand-Object Interaction Pretraining from Videoscited by 39
  29. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 36
  30. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
  31. 2022A Real2Sim2Real Method for Robust Object Grasping with Neural Surface Reconstructioncited by 19
  32. 2024Deep SE(3)-Equivariant Geometric Reasoning for Precise Placement Taskscited by 19
  33. 2024ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decompositioncited by 14
  34. 2023Dexterous Pre-grasp Manipulation for Human-like Functional Categorical Grasping: Deep Reinforcement Learning and Grasp Representationscited by 4
  35. 2025Soft and Compliant Contact-Rich Hair Manipulation and Carecited by 2
  36. 2024Functional Eigen-Grasping Using Approach Heatmapscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  2. 2026Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstructioncited by 11
  3. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 10
  4. 2025TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactionscited by 8
  5. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 7
  6. 2026Robot Learning from Human Videos: A Surveycited by 5
  7. 2026Human Universal Graspingcited by 5
  8. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 3
  9. 2026Generate, Transfer, Adapt: Learning Functional Dexterous Grasping from a Single Human Demonstrationcited by 1
  10. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 1
  11. 2026ConGraspXL: Controllable Constraint-Conditioned Dexterous Grasping Motion Synthesiscited by 1