# Simultaneous Estimation of Hand Configurations and Finger Joint Angles using Forearm Ultrasound

arXiv: https://arxiv.org/abs/2211.15871 · HTML: https://mlsanity.com/paper/2211.15871/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 72 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 5 of 5)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Prediction of Metacarpophalangeal joint angles and Classification of Hand configurations based on Ultrasound Imaging of the Forearm](https://mlsanity.com/paper/2109.11093/) (2021)

## Built on by (most cited first, 7 of 7)

- [Forearm Ultrasound based Gesture Recognition on Edge](https://mlsanity.com/paper/2409.09915/) (2024)
- [GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM](https://mlsanity.com/paper/2407.10870/) (2024)
- [Random Channel Ablation for Robust Hand Gesture Classification with Multimodal Biosignals](https://mlsanity.com/paper/2407.10874/) (2024)
- [Improving Intersession Reproducibility for Forearm Ultrasound based Hand Gesture Classification through an Incremental Learning Approach](https://mlsanity.com/paper/2409.16415/) (2024)
- [Hand Gesture Classification Based on Forearm Ultrasound Video Snippets Using 3D Convolutional Neural Networks](https://mlsanity.com/paper/2409.16431/) (2024)
- [Simultaneous Estimation of Manipulation Skill and Hand Grasp Force from Forearm Ultrasound Images](https://mlsanity.com/paper/2502.00275/) (2025)
- [SonoRank: Towards Calibration-Free Real-Time Finger Flexion Detection from Forearm Ultrasound Sequences](https://mlsanity.com/paper/2607.07542/) (2026)

## Abstract (arXiv)

With the advancement in computing and robotics, it is necessary to develop fluent and intuitive methods for interacting with digital systems, augmented/virtual reality (AR/VR) interfaces, and physical robotic systems. Hand motion recognition is widely used to enable these interactions. Hand configuration classification and MCP joint angle detection is important for a comprehensive reconstruction of hand motion. sEMG and other technologies have been used for the detection of hand motions. Forearm ultrasound images provide a musculoskeletal visualization that can be used to understand hand motion. Recent work has shown that these ultrasound images can be classified using machine learning to estimate discrete hand configurations. Estimating both hand configuration and MCP joint angles based on forearm ultrasound has not been addressed in the literature. In this paper, we propose a CNN based deep learning pipeline for predicting the MCP joint angles. The results for the hand configuration classification were compared by using different machine learning algorithms. SVC with different kernels, MLP, and the proposed CNN have been used to classify the ultrasound images into 11 hand configurations based on activities of daily living. Forearm ultrasound images were acquired from 6 subjects instructed to move their hands according to predefined hand configurations. Motion capture data was acquired to get the finger angles corresponding to the hand movements at different speeds. Average classification accuracy of 82.7% for the proposed CNN and over 80% for SVC for different kernels was observed on a subset of the dataset. An average RMSE of 7.35 degrees was obtained between the predicted and the true MCP joint angles. A low latency (6.25 - 9.1 Hz) pipeline has been proposed for estimating both MCP joint angles and hand configuration aimed at real-time control of human-machine interfaces.