# Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interaction

arXiv: https://arxiv.org/abs/2204.13998 · HTML: https://mlsanity.com/paper/2204.13998/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 74 references that are in the map.
- First posted on arXiv 3 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations](https://mlsanity.com/paper/1912.04344/) (2019)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)
- [Shape Completion Enabled Robotic Grasping](https://mlsanity.com/paper/1609.08546/) (2016)
- [Deep Differentiable Grasp Planner for High-DOF Grippers](https://mlsanity.com/paper/2002.01530/) (2020)
- [Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation](https://mlsanity.com/paper/1710.06422/) (2017)
- [High precision grasp pose detection in dense clutter](https://mlsanity.com/paper/1603.01564/) (2016)
- [Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods](https://mlsanity.com/paper/1802.10264/) (2018)
- [Learning a visuomotor controller for real world robotic grasping using simulated depth images](https://mlsanity.com/paper/1706.04652/) (2017)
- [Learning Continuous 3D Reconstructions for Geometrically Aware Grasping](https://mlsanity.com/paper/1910.00983/) (2019)
- [Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Network](https://mlsanity.com/paper/1804.03289/) (2018)
- [Generating Grasp Poses for a High-DOF Gripper Using Neural Networks](https://mlsanity.com/paper/1903.00425/) (2019)
- [Manipulation Trajectory Optimization with Online Grasp Synthesis and Selection](https://mlsanity.com/paper/1911.10280/) (2019)
- [AdaGrasp: Learning an Adaptive Gripper-Aware Grasping Policy](https://mlsanity.com/paper/2011.14206/) (2020)
- [Understanding and Exploiting Object Interaction Landscapes](https://mlsanity.com/paper/1609.08685/) (2016)
- [New Formulation of Mixed-Integer Conic Programming for Globally Optimal Grasp Planning](https://mlsanity.com/paper/1909.05430/) (2019)

## Built on by (most cited first, 24 of 24)

- [UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy](https://mlsanity.com/paper/2303.00938/) (2023)
- [UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning](https://mlsanity.com/paper/2304.00464/) (2023)
- [DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions](https://mlsanity.com/paper/2403.17827/) (2024)
- [GraspXL: Generating Grasping Motions for Diverse Objects at Scale](https://mlsanity.com/paper/2403.19649/) (2024)
- [GenDexGrasp: Generalizable Dexterous Grasping](https://mlsanity.com/paper/2210.00722/) (2022)
- [CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesis](https://mlsanity.com/paper/2303.15469/) (2023)
- [ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation](https://mlsanity.com/paper/2309.03891/) (2023)
- [RealDex: Towards Human-like Grasping for Robotic Dexterous Hand](https://mlsanity.com/paper/2402.13853/) (2024)
- [GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities](https://mlsanity.com/paper/2412.04244/) (2024)
- [GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency](https://mlsanity.com/paper/2308.11617/) (2023)
- [GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping](https://mlsanity.com/paper/2309.06038/) (2023)
- [SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation](https://mlsanity.com/paper/2310.16838/) (2023)
- [Grasp Multiple Objects with One Hand](https://mlsanity.com/paper/2310.15599/) (2023)
- [HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects](https://mlsanity.com/paper/2301.07652/) (2023)
- [Learning to Transfer In-Hand Manipulations Using a Greedy Shape Curriculum](https://mlsanity.com/paper/2303.12726/) (2023)
- [Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping](https://mlsanity.com/paper/2410.23039/) (2024)
- [G-DexGrasp: Generalizable Dexterous Grasping Synthesis Via Part-Aware Prior Retrieval and Prior-Assisted Generation](https://mlsanity.com/paper/2503.19457/) (2025)
- [NIFT: Neural Interaction Field and Template for Object Manipulation](https://mlsanity.com/paper/2210.10992/) (2022)
- [Reconstructing 3D Human Pose from RGB-D Data with Occlusions](https://mlsanity.com/paper/2310.01228/) (2023)
- [UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis](https://mlsanity.com/paper/2607.28198/) (2026)
- [DMFC-GraspNet: Differentiable Multi-Fingered Robotic Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2308.00456/) (2023)
- [Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis](https://mlsanity.com/paper/2609.04096/) (2026)
- [PartHOI: Part-based Hand-Object Interaction Transfer via Generalized Cylinders](https://mlsanity.com/paper/2504.20599/) (2025)
- [Grasp2Twist: Learning Bimanual Dexterous Jar Opening by Reinforcement Learning](https://mlsanity.com/paper/2609.32064/) (2026)

## Abstract (arXiv)

We approach the problem of high-DOF reaching-and-grasping via learning joint planning of grasp and motion with deep reinforcement learning. To resolve the sample efficiency issue in learning the high-dimensional and complex control of dexterous grasping, we propose an effective representation of grasping state characterizing the spatial interaction between the gripper and the target object. To represent gripper-object interaction, we adopt Interaction Bisector Surface (IBS) which is the Voronoi diagram between two close by 3D geometric objects and has been successfully applied in characterizing spatial relations between 3D objects. We found that IBS is surprisingly effective as a state representation since it well informs the fine-grained control of each finger with spatial relation against the target object. This novel grasp representation, together with several technical contributions including a fast IBS approximation, a novel vector-based reward and an effective training strategy, facilitate learning a strong control model of high-DOF grasping with good sample efficiency, dynamic adaptability, and cross-category generality. Experiments show that it generates high-quality dexterous grasp for complex shapes with smooth grasping motions.