# Combining Vision and EMG-Based Hand Tracking for Extended Reality Musical Instruments

arXiv: https://arxiv.org/abs/2307.10203 · HTML: https://mlsanity.com/paper/2307.10203/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 26 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 2 of 2)

- [putEMG -- a surface electromyography hand gesture recognition dataset](https://mlsanity.com/paper/1907.07428/) (2019)
- [Problems and Prospects for Intimate Musical Control of Computers](https://mlsanity.com/paper/2010.01570/) (2020)

## Abstract (arXiv)

Hand tracking is a critical component of natural user interactions in extended reality (XR) environments, including extended reality musical instruments (XRMIs). However, self-occlusion remains a significant challenge for vision-based hand tracking systems, leading to inaccurate results and degraded user experiences. In this paper, we propose a multimodal hand tracking system that combines vision-based hand tracking with surface electromyography (sEMG) data for finger joint angle estimation. We validate the effectiveness of our system through a series of hand pose tasks designed to cover a wide range of gestures, including those prone to self-occlusion. By comparing the performance of our multimodal system to a baseline vision-based tracking method, we demonstrate that our multimodal approach significantly improves tracking accuracy for several finger joints prone to self-occlusion. These findings suggest that our system has the potential to enhance XR experiences by providing more accurate and robust hand tracking, even in the presence of self-occlusion.