# Dexterous Functional Grasping

arXiv: https://arxiv.org/abs/2312.02975 · HTML: https://mlsanity.com/paper/2312.02975/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 42)

- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality](https://mlsanity.com/paper/2210.13702/) (2022)
- [ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging](https://mlsanity.com/paper/1904.06830/) (2019)
- [ContactOpt: Optimizing Contact to Improve Grasps](https://mlsanity.com/paper/2104.07267/) (2021)
- [Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior](https://mlsanity.com/paper/2212.03238/) (2022)
- [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022)
- [UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy](https://mlsanity.com/paper/2303.00938/) (2023)
- [In-Hand Object Rotation via Rapid Motor Adaptation](https://mlsanity.com/paper/2210.04887/) (2022)
- [DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System](https://mlsanity.com/paper/1910.03135/) (2019)
- [Affordance Diffusion: Synthesizing Hand-Object Interactions](https://mlsanity.com/paper/2303.12538/) (2023)
- [VideoDex: Learning Dexterity from Internet Videos](https://mlsanity.com/paper/2212.04498/) (2022)

## Built on by (most cited first, 30 of 39)

- [3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations](https://mlsanity.com/paper/2403.03954/) (2024)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025)
- [GraspXL: Generating Grasping Motions for Diverse Objects at Scale](https://mlsanity.com/paper/2403.19649/) (2024)
- [DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes](https://mlsanity.com/paper/2410.23004/) (2024)
- [Hand-Object Interaction Pretraining from Videos](https://mlsanity.com/paper/2409.08273/) (2024)
- [World Models for Learning Dexterous Hand-Object Interactions from Human Videos](https://mlsanity.com/paper/2512.13644/) (2025)
- [Cross-Embodiment Dexterous Grasping with Reinforcement Learning](https://mlsanity.com/paper/2410.02479/) (2024)
- [Lessons from Learning to Spin "Pens"](https://mlsanity.com/paper/2407.18902/) (2024)
- [Bimanual Dexterity for Complex Tasks](https://mlsanity.com/paper/2411.13677/) (2024)
- [Local Policies Enable Zero-shot Long-horizon Manipulation](https://mlsanity.com/paper/2410.22332/) (2024)
- [Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy](https://mlsanity.com/paper/2504.18829/) (2025)
- [DexVLG: Dexterous Vision-Language-Grasp Model at Scale](https://mlsanity.com/paper/2507.02747/) (2025)
- [SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation](https://mlsanity.com/paper/2602.16863/) (2026)
- [ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decomposition](https://mlsanity.com/paper/2403.18062/) (2024)
- [Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation](https://mlsanity.com/paper/2602.15828/) (2026)
- [Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions](https://mlsanity.com/paper/2505.05517/) (2025)
- [Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction](https://mlsanity.com/paper/2602.09013/) (2026)
- [Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping](https://mlsanity.com/paper/2410.02475/) (2024)
- [CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion](https://mlsanity.com/paper/2506.14769/) (2025)
- [Dexterous Functional Pre-Grasp Manipulation with Diffusion Policy](https://mlsanity.com/paper/2403.12421/) (2024)
- [Universal Dexterous Functional Grasping via Demonstration-Editing Reinforcement Learning](https://mlsanity.com/paper/2512.13380/) (2025)
- [DextER: Language-driven Dexterous Grasp Generation with Embodied Reasoning](https://mlsanity.com/paper/2601.16046/) (2026)
- [Afford-X: Generalizable and Slim Affordance Reasoning for Task-oriented Manipulation](https://mlsanity.com/paper/2503.03556/) (2025)
- [Contact Map Transfer with Conditional Diffusion Model for Generalizable Dexterous Grasp Generation](https://mlsanity.com/paper/2511.01276/) (2025)
- [PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generation](https://mlsanity.com/paper/2603.22193/) (2026)
- [Grasp2Grasp: Vision-Based Dexterous Grasp Translation via Schr"odinger Bridges](https://mlsanity.com/paper/2506.02489/) (2025)
- [Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress](https://mlsanity.com/paper/2509.24129/) (2025)
- [OmniDexVLG: Learning Dexterous Grasp Generation from Vision Language Model-Guided Grasp Semantics, Taxonomy and Functional Affordance](https://mlsanity.com/paper/2512.03874/) (2025)
- [Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos](https://mlsanity.com/paper/2602.13197/) (2026)

## Abstract (arXiv)

While there have been significant strides in dexterous manipulation, most of it is limited to benchmark tasks like in-hand reorientation which are of limited utility in the real world. The main benefit of dexterous hands over two-fingered ones is their ability to pickup tools and other objects (including thin ones) and grasp them firmly to apply force. However, this task requires both a complex understanding of functional affordances as well as precise low-level control. While prior work obtains affordances from human data this approach doesn't scale to low-level control. Similarly, simulation training cannot give the robot an understanding of real-world semantics. In this paper, we aim to combine the best of both worlds to accomplish functional grasping for in-the-wild objects. We use a modular approach. First, affordances are obtained by matching corresponding regions of different objects and then a low-level policy trained in sim is run to grasp it. We propose a novel application of eigengrasps to reduce the search space of RL using a small amount of human data and find that it leads to more stable and physically realistic motion. We find that eigengrasp action space beats baselines in simulation and outperforms hardcoded grasping in real and matches or outperforms a trained human teleoperator. Results visualizations and videos at https://dexfunc.github.io/