# From Hand-Perspective Visual Information to Grasp Type Probabilities: Deep Learning via Ranking Labels

arXiv: https://arxiv.org/abs/2103.04863 · HTML: https://mlsanity.com/paper/2103.04863/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 33 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Visual Cues to Improve Myoelectric Control of Upper Limb Prostheses](https://mlsanity.com/paper/1709.02236/) (2017)
- [Hierarchical Graphical Models for Context-Aware Hybrid Brain-Machine Interfaces](https://mlsanity.com/paper/1809.05635/) (2018)

## Built on by (most cited first, 2 of 2)

- [Inference of Upcoming Human Grasp Using EMG During Reach-to-Grasp Movement](https://mlsanity.com/paper/2104.09627/) (2021)
- [DcnnGrasp: Towards Accurate Grasp Pattern Recognition with Adaptive Regularizer Learning](https://mlsanity.com/paper/2205.05218/) (2022)

## Abstract (arXiv)

Limb deficiency severely affects the daily lives of amputees and drives efforts to provide functional robotic prosthetic hands to compensate this deprivation. Convolutional neural network-based computer vision control of the prosthetic hand has received increased attention as a method to replace or complement physiological signals due to its reliability by training visual information to predict the hand gesture. Mounting a camera into the palm of a prosthetic hand is proved to be a promising approach to collect visual data. However, the grasp type labelled from the eye and hand perspective may differ as object shapes are not always symmetric. Thus, to represent this difference in a realistic way, we employed a dataset containing synchronous images from eye- and hand- view, where the hand-perspective images are used for training while the eye-view images are only for manual labelling. Electromyogram (EMG) activity and movement kinematics data from the upper arm are also collected for multi-modal information fusion in future work. Moreover, in order to include human-in-the-loop control and combine the computer vision with physiological signal inputs, instead of making absolute positive or negative predictions, we build a novel probabilistic classifier according to the Plackett-Luce model. To predict the probability distribution over grasps, we exploit the statistical model over label rankings to solve the permutation domain problems via a maximum likelihood estimation, utilizing the manually ranked lists of grasps as a new form of label. We indicate that the proposed model is applicable to the most popular and productive convolutional neural network frameworks.