# Transformer-based Hand Gesture Recognition via High-Density EMG Signals: From Instantaneous Recognition to Fusion of Motor Unit Spike Trains

arXiv: https://arxiv.org/abs/2212.00743 · HTML: https://mlsanity.com/paper/2212.00743/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 50 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning](https://mlsanity.com/paper/1801.07756/) (2018)
- [FS-HGR: Few-shot Learning for Hand Gesture Recognition via ElectroMyography](https://mlsanity.com/paper/2011.06104/) (2020)
- [Domain Adaptation for sEMG-based Gesture Recognition with Recurrent Neural Networks](https://mlsanity.com/paper/1901.06958/) (2019)
- [TEMGNet: Deep Transformer-based Decoding of Upperlimb sEMG for Hand Gestures Recognition](https://mlsanity.com/paper/2109.12379/) (2021)
- [Hand Gesture Recognition Using Temporal Convolutions and Attention Mechanism](https://mlsanity.com/paper/2110.08717/) (2021)
- [ViT-HGR: Vision Transformer-based Hand Gesture Recognition from High Density Surface EMG Signals](https://mlsanity.com/paper/2201.10060/) (2022)
- [Real-Time EMG Signal Classification via Recurrent Neural Networks](https://mlsanity.com/paper/2109.05674/) (2021)

## Abstract (arXiv)

Designing efficient and labor-saving prosthetic hands requires powerful hand gesture recognition algorithms that can achieve high accuracy with limited complexity and latency. In this context, the paper proposes a compact deep learning framework referred to as the CT-HGR, which employs a vision transformer network to conduct hand gesture recognition using highdensity sEMG (HD-sEMG) signals. The attention mechanism in the proposed model identifies similarities among different data segments with a greater capacity for parallel computations and addresses the memory limitation problems while dealing with inputs of large sequence lengths. CT-HGR can be trained from scratch without any need for transfer learning and can simultaneously extract both temporal and spatial features of HD-sEMG data. Additionally, the CT-HGR framework can perform instantaneous recognition using sEMG image spatially composed from HD-sEMG signals. A variant of the CT-HGR is also designed to incorporate microscopic neural drive information in the form of Motor Unit Spike Trains (MUSTs) extracted from HD-sEMG signals using Blind Source Separation (BSS). This variant is combined with its baseline version via a hybrid architecture to evaluate potentials of fusing macroscopic and microscopic neural drive information. The utilized HD-sEMG dataset involves 128 electrodes that collect the signals related to 65 isometric hand gestures of 20 subjects. The proposed CT-HGR framework is applied to 31.25, 62.5, 125, 250 ms window sizes of the above-mentioned dataset utilizing 32, 64, 128 electrode channels. The average accuracy over all the participants using 32 electrodes and a window size of 31.25 ms is 86.23%, which gradually increases till reaching 91.98% for 128 electrodes and a window size of 250 ms. The CT-HGR achieves accuracy of 89.13% for instantaneous recognition based on a single frame of HD-sEMG image.