# Multiscaled Multi-Head Attention-based Video Transformer Network for Hand Gesture Recognition

arXiv: https://arxiv.org/abs/2501.00935 · HTML: https://mlsanity.com/paper/2501.00935/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 1 Jan 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition](https://mlsanity.com/paper/2008.09412/) (2020)
- [Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks](https://mlsanity.com/paper/1901.10323/) (2019)
- [Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training](https://mlsanity.com/paper/1812.06145/) (2018)
- [A robust and efficient video representation for action recognition](https://mlsanity.com/paper/1504.05524/) (2015)
- [Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition](https://mlsanity.com/paper/1804.07187/) (2018)
- [Res3ATN -- Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos](https://mlsanity.com/paper/2001.01083/) (2020)
- [Multi-modal Fusion for Single-Stage Continuous Gesture Recognition](https://mlsanity.com/paper/2011.04945/) (2020)
- [Exploiting Recurrent Neural Networks and Leap Motion Controller for Sign Language and Semaphoric Gesture Recognition](https://mlsanity.com/paper/1803.10435/) (2018)
- [Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition](https://mlsanity.com/paper/2110.15639/) (2021)

## Built on by (most cited first, 2 of 2)

- [UniCon-Former: Unified Convolution Transformer is All You Need for Hand Gesture Recognition](https://mlsanity.com/paper/2608.13217/) (2026)
- [SoftModel: A Neural Model That Grows Its Own Topology -- Governed Structural Growth for Continual In-Service Learning](https://mlsanity.com/paper/2608.16409/) (2026)

## Abstract (arXiv)

Dynamic gesture recognition is one of the challenging research areas due to variations in pose, size, and shape of the signer's hand. In this letter, Multiscaled Multi-Head Attention Video Transformer Network (MsMHA-VTN) for dynamic hand gesture recognition is proposed. A pyramidal hierarchy of multiscale features is extracted using the transformer multiscaled head attention model. The proposed model employs different attention dimensions for each head of the transformer which enables it to provide attention at the multiscale level. Further, in addition to single modality, recognition performance using multiple modalities is examined. Extensive experiments demonstrate the superior performance of the proposed MsMHA-VTN with an overall accuracy of 88.22% and 99.10% on NVGesture and Briareo datasets, respectively.