# Towards Predicting Fine Finger Motions from Ultrasound Images via Kinematic Representation

arXiv: https://arxiv.org/abs/2202.05204 · HTML: https://mlsanity.com/paper/2202.05204/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 10 Feb 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [Recent Advances in Convolutional Neural Networks](https://mlsanity.com/paper/1512.07108/) (2015)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks](https://mlsanity.com/paper/1903.02193/) (2019)
- [Deep learning in ultrasound imaging](https://mlsanity.com/paper/1907.02994/) (2019)
- [A Two-stage Unsupervised Approach for Low light Image Enhancement](https://mlsanity.com/paper/2010.09316/) (2020)
- [Towards CT-quality Ultrasound Imaging using Deep Learning](https://mlsanity.com/paper/1710.06304/) (2017)
- [Prediction of Metacarpophalangeal joint angles and Classification of Hand configurations based on Ultrasound Imaging of the Forearm](https://mlsanity.com/paper/2109.11093/) (2021)
- [Learning beamforming in ultrasound imaging](https://mlsanity.com/paper/1812.08043/) (2018)
- [Intuitive Neuromyoelectric Control of a Dexterous Bionic Arm Using a Modified Kalman Filter](https://mlsanity.com/paper/1908.10522/) (2019)

## Built on by (most cited first, 1 of 1)

- [SonoRank: Towards Calibration-Free Real-Time Finger Flexion Detection from Forearm Ultrasound Sequences](https://mlsanity.com/paper/2607.07542/) (2026)

## Abstract (arXiv)

A central challenge in building robotic prostheses is the creation of a sensor-based system able to read physiological signals from the lower limb and instruct a robotic hand to perform various tasks. Existing systems typically perform discrete gestures such as pointing or grasping, by employing electromyography (EMG) or ultrasound (US) technologies to analyze muscle states. While estimating finger gestures has been done in the past by detecting prominent gestures, we are interested in detection, or inference, done in the context of fine motions that evolve over time. Examples include motions occurring when performing fine and dexterous tasks such as keyboard typing or piano playing. We consider this task as an important step towards higher adoption rates of robotic prostheses among arm amputees, as it has the potential to dramatically increase functionality in performing daily tasks. To this end, we present an end-to-end robotic system, which can successfully infer fine finger motions. This is achieved by modeling the hand as a robotic manipulator and using it as an intermediate representation to encode muscles' dynamics from a sequence of US images. We evaluated our method by collecting data from a group of subjects and demonstrating how it can be used to replay music played or text typed. To the best of our knowledge, this is the first study demonstrating these downstream tasks within an end-to-end system.