# Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

arXiv: https://arxiv.org/abs/2203.09812 · HTML: https://mlsanity.com/paper/2203.09812/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 8 of 8)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control](https://mlsanity.com/paper/2104.03893/) (2021)
- [Towards Creating a Deployable Grasp Type Probability Estimator for a Prosthetic Hand](https://mlsanity.com/paper/2101.05357/) (2021)
- [Using Synthetic Data and Deep Networks to Recognize Primitive Shapes for Object Grasping](https://mlsanity.com/paper/1909.08508/) (2019)
- [A Soft Humanoid Hand with In-Finger Visual Perception](https://mlsanity.com/paper/2006.03537/) (2020)

## Built on by (most cited first, 6 of 6)

- [A Powered Prosthetic Hand with Vision System for Enhancing the Anthropopathic Grasp](https://mlsanity.com/paper/2412.07105/) (2024)
- [Continuous Wrist Control on the Hannes Prosthesis: a Vision-based Shared Autonomy Framework](https://mlsanity.com/paper/2502.17265/) (2025)
- [Bring Your Own Grasp Generator: Leveraging Robot Grasp Generation for Prosthetic Grasping](https://mlsanity.com/paper/2503.00466/) (2025)
- [Semi-autonomous Prosthesis Control Using Minimal Depth Information and Vibrotactile Feedback](https://mlsanity.com/paper/2210.00541/) (2022)
- [HannesImitation: Grasping with the Hannes Prosthetic Hand via Imitation Learning](https://mlsanity.com/paper/2508.00491/) (2025)
- [Grasp-HGN: Grasping the Unexpected](https://mlsanity.com/paper/2508.07648/) (2025)

## Abstract (arXiv)

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-in-hand systems automatically control the hand pre-shaping before the grasp, based on visual input coming from a camera on the wrist. In this paper, we present an eye-in-hand learning-based approach for hand pre-shape classification from RGB sequences. Differently from previous work, we design the system to support the possibility to grasp each considered object part with a different grasp type. In order to overcome the lack of data of this kind and reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. We make publicly available the code and dataset to reproduce the presented results.