# Capturing complex hand movements and object interactions using machine learning-powered stretchable smart textile gloves

arXiv: https://arxiv.org/abs/2410.02221 · HTML: https://mlsanity.com/paper/2410.02221/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 51 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [Physics-based Human Motion Estimation and Synthesis from Videos](https://mlsanity.com/paper/2109.09913/) (2021)
- [Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning](https://mlsanity.com/paper/1801.07756/) (2018)
- [Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects](https://mlsanity.com/paper/2003.12045/) (2020)
- [An EMG Gesture Recognition System with Flexible High-Density Sensors and Brain-Inspired High-Dimensional Classifier](https://mlsanity.com/paper/1802.10237/) (2018)
- [Dynamic Modeling of Hand-Object Interactions via Tactile Sensing](https://mlsanity.com/paper/2109.04378/) (2021)
- [Intelligent Knee Sleeves: A Real-time Multimodal Dataset for 3D Lower Body Motion Estimation Using Smart Textile](https://mlsanity.com/paper/2311.12829/) (2023)

## Built on by (most cited first, 5 of 5)

- [emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation](https://mlsanity.com/paper/2412.02725/) (2024)
- [A Time-Series Data Augmentation Model through Diffusion and Transformer Integration](https://mlsanity.com/paper/2505.03790/) (2025)
- [Position: Olfaction Standardization is Essential for the Advancement of Embodied Artificial Intelligence](https://mlsanity.com/paper/2506.00398/) (2025)
- [Visuo-Acoustic Hand Pose and Contact Estimation](https://mlsanity.com/paper/2508.00852/) (2025)
- [AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Gloves](https://mlsanity.com/paper/2602.05159/) (2026)

## Abstract (arXiv)

Accurate real-time tracking of dexterous hand movements and interactions has numerous applications in human-computer interaction, metaverse, robotics, and tele-health. Capturing realistic hand movements is challenging because of the large number of articulations and degrees of freedom. Here, we report accurate and dynamic tracking of articulated hand and finger movements using stretchable, washable smart gloves with embedded helical sensor yarns and inertial measurement units. The sensor yarns have a high dynamic range, responding to low 0.005 % to high 155 % strains, and show stability during extensive use and washing cycles. We use multi-stage machine learning to report average joint angle estimation root mean square errors of 1.21 and 1.45 degrees for intra- and inter-subjects cross-validation, respectively, matching accuracy of costly motion capture cameras without occlusion or field of view limitations. We report a data augmentation technique that enhances robustness to noise and variations of sensors. We demonstrate accurate tracking of dexterous hand movements during object interactions, opening new avenues of applications including accurate typing on a mock paper keyboard, recognition of complex dynamic and static gestures adapted from American Sign Language and object identification.