# Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Network

arXiv: https://arxiv.org/abs/1804.03289 · HTML: https://mlsanity.com/paper/1804.03289/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2018.

## Built on by (most cited first, 19 of 19)

- [SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion](https://mlsanity.com/paper/2209.03855/) (2022)
- [UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning](https://mlsanity.com/paper/2304.00464/) (2023)
- [DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation](https://mlsanity.com/paper/2211.09423/) (2022)
- [Learning Generalizable Dexterous Manipulation from Human Grasp Affordance](https://mlsanity.com/paper/2204.02320/) (2022)
- [Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations](https://mlsanity.com/paper/2207.05053/) (2022)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interaction](https://mlsanity.com/paper/2204.13998/) (2022)
- [SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation](https://mlsanity.com/paper/2310.16838/) (2023)
- [Keypoint-GraspNet: Keypoint-based 6-DoF Grasp Generation from the Monocular RGB-D input](https://mlsanity.com/paper/2209.08752/) (2022)
- [Where To Start? Transferring Simple Skills to Complex Environments](https://mlsanity.com/paper/2212.06111/) (2022)
- [GKNet: grasp keypoint network for grasp candidates detection](https://mlsanity.com/paper/2106.08497/) (2021)
- [Simulation-based Bayesian inference for multi-fingered robotic grasping](https://mlsanity.com/paper/2109.14275/) (2021)
- [KGNv2: Separating Scale and Pose Prediction for Keypoint-based 6-DoF Grasp Synthesis on RGB-D input](https://mlsanity.com/paper/2303.05617/) (2023)
- [Simulation-based Bayesian inference for robotic grasping](https://mlsanity.com/paper/2303.05873/) (2023)
- [DMFC-GraspNet: Differentiable Multi-Fingered Robotic Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2308.00456/) (2023)
- [Learning 6DoF Grasping Using Reward-Consistent Demonstration](https://mlsanity.com/paper/2103.12321/) (2021)
- [Implicit representation priors meet Riemannian geometry for Bayesian robotic grasping](https://mlsanity.com/paper/2304.08805/) (2023)
- [LensDFF: Language-enhanced Sparse Feature Distillation for Efficient Few-Shot Dexterous Manipulation](https://mlsanity.com/paper/2503.03890/) (2025)
- [Constrained Stein Variational Gradient Descent for Robot Perception, Planning, and Identification](https://mlsanity.com/paper/2506.00589/) (2025)

## Abstract (arXiv)

We propose a novel approach to multi-fingered grasp planning leveraging learned deep neural network models. We train a convolutional neural network to predict grasp success as a function of both visual information of an object and grasp configuration. We can then formulate grasp planning as inferring the grasp configuration which maximizes the probability of grasp success. We efficiently perform this inference using a gradient-ascent optimization inside the neural network using the backpropagation algorithm. Our work is the first to directly plan high quality multifingered grasps in configuration space using a deep neural network without the need of an external planner. We validate our inference method performing both multifinger and two-finger grasps on real robots. Our experimental results show that our planning method outperforms existing planning methods for neural networks; while offering several other benefits including being data-efficient in learning and fast enough to be deployed in real robotic applications.