# Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control arXiv: https://arxiv.org/abs/2104.03893 ยท HTML: https://mlsanity.com/paper/2104.03893/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 52 references that are in the map. - First posted on arXiv 8 Apr 2021. ## Builds on (most cited first, 13 of 13) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020) - [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020) - [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018) - [Synthesizing Training Data for Object Detection in Indoor Scenes](https://mlsanity.com/paper/1702.07836/) (2017) - [Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction](https://mlsanity.com/paper/1405.0006/) (2014) - [Greedy Gaussian Segmentation of Multivariate Time Series](https://mlsanity.com/paper/1610.07435/) (2016) - [Towards Creating a Deployable Grasp Type Probability Estimator for a Prosthetic Hand](https://mlsanity.com/paper/2101.05357/) (2021) - [Egocentric Gesture Recognition for Head-Mounted AR devices](https://mlsanity.com/paper/1808.05380/) (2018) - [HANDS: A Multimodal Dataset for Modeling Towards Human Grasp Intent Inference in Prosthetic Hands](https://mlsanity.com/paper/2103.04845/) (2021) - [A novel approach to classify natural grasp actions by estimating muscle activity patterns from EEG signals](https://mlsanity.com/paper/2002.00556/) (2020) ## Built on by (most cited first, 13 of 13) - [Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis](https://mlsanity.com/paper/2203.09812/) (2022) - [A Powered Prosthetic Hand with Vision System for Enhancing the Anthropopathic Grasp](https://mlsanity.com/paper/2412.07105/) (2024) - [EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation](https://mlsanity.com/paper/2605.05712/) (2026) - [EMAHA-DB1: A New Upper Limb sEMG Dataset for Classification of Activities of Daily Living](https://mlsanity.com/paper/2301.03325/) (2023) - [Towards Biosignals-Free Autonomous Prosthetic Hand Control via Imitation Learning](https://mlsanity.com/paper/2506.08795/) (2025) - [HannesImitation: Grasping with the Hannes Prosthetic Hand via Imitation Learning](https://mlsanity.com/paper/2508.00491/) (2025) - [A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot](https://mlsanity.com/paper/2508.14994/) (2025) - [Inference of Upcoming Human Grasp Using EMG During Reach-to-Grasp Movement](https://mlsanity.com/paper/2104.09627/) (2021) - [Fuzzy Expert Systems for Prediction of ICU Admission in Patients with COVID-19](https://mlsanity.com/paper/2104.12868/) (2021) - [DcnnGrasp: Towards Accurate Grasp Pattern Recognition with Adaptive Regularizer Learning](https://mlsanity.com/paper/2205.05218/) (2022) - [Grasp-HGN: Grasping the Unexpected](https://mlsanity.com/paper/2508.07648/) (2025) - [BIONIX: A Wireless, Low-Cost Prosthetic Arm with Dual-Signal EEG and EMG Control](https://mlsanity.com/paper/2512.16929/) (2025) - [EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimation](https://mlsanity.com/paper/2610.06970/) (2026) ## Abstract (arXiv) Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities. Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components. Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%. Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time.