# Skeleton-based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module

arXiv: https://arxiv.org/abs/1811.07081 · HTML: https://mlsanity.com/paper/1811.07081/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2018.

## Built on by (most cited first, 13 of 13)

- [Logsig-RNN: a novel network for robust and efficient skeleton-based action recognition](https://mlsanity.com/paper/2110.13008/) (2021)
- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval](https://mlsanity.com/paper/2407.16394/) (2024)
- [ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos](https://mlsanity.com/paper/2108.10059/) (2021)
- [Multi-Modal Zero-Shot Sign Language Recognition](https://mlsanity.com/paper/2109.00796/) (2021)
- [GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2403.15212/) (2024)
- [Dual Attention Network for Heart Rate and Respiratory Rate Estimation](https://mlsanity.com/paper/2111.00390/) (2021)
- [Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation](https://mlsanity.com/paper/2405.08969/) (2024)
- [Online hand gesture recognition using Continual Graph Transformers](https://mlsanity.com/paper/2502.14939/) (2025)
- [Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperation](https://mlsanity.com/paper/2109.08263/) (2021)
- [SHARDeg: A Benchmark for Skeletal Human Action Recognition in Degraded Scenarios](https://mlsanity.com/paper/2505.18048/) (2025)
- [OMG-Bench: A New Challenging Benchmark for Skeleton-based Online Micro Hand Gesture Recognition](https://mlsanity.com/paper/2512.16727/) (2025)
- [ByteAction: Byte-space Action Recognition Foundation Model](https://mlsanity.com/paper/2608.22760/) (2026)

## Abstract (arXiv)

The skeleton based gesture recognition is gaining more popularity due to its wide possible applications. The key issues are how to extract discriminative features and how to design the classification model. In this paper, we first leverage a robust feature descriptor, path signature (PS), and propose three PS features to explicitly represent the spatial and temporal motion characteristics, i.e., spatial PS (S_PS), temporal PS (T_PS) and temporal spatial PS (T_S_PS). Considering the significance of fine hand movements in the gesture, we propose an "attention on hand" (AOH) principle to define joint pairs for the S_PS and select single joint for the T_PS. In addition, the dyadic method is employed to extract the T_PS and T_S_PS features that encode global and local temporal dynamics in the motion. Secondly, without the recurrent strategy, the classification model still faces challenges on temporal variation among different sequences. We propose a new temporal transformer module (TTM) that can match the sequence key frames by learning the temporal shifting parameter for each input. This is a learning-based module that can be included into standard neural network architecture. Finally, we design a multi-stream fully connected layer based network to treat spatial and temporal features separately and fused them together for the final result. We have tested our method on three benchmark gesture datasets, i.e., ChaLearn 2016, ChaLearn 2013 and MSRC-12. Experimental results demonstrate that we achieve the state-of-the-art performance on skeleton-based gesture recognition with high computational efficiency.