# GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition

arXiv: https://arxiv.org/abs/2405.11180 · HTML: https://mlsanity.com/paper/2405.11180/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 72 references that are in the map.
- First posted on arXiv 18 May 2024.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [Patches Are All You Need?](https://mlsanity.com/paper/2201.09792/) (2022)
- [UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2](https://mlsanity.com/paper/2012.03539/) (2020)
- [CenterFormer: Center-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2209.05588/) (2022)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [P2T: Pyramid Pooling Transformer for Scene Understanding](https://mlsanity.com/paper/2106.12011/) (2021)
- [Vision Transformer for Fast and Efficient Scene Text Recognition](https://mlsanity.com/paper/2105.08582/) (2021)
- [POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition](https://mlsanity.com/paper/2204.04083/) (2022)
- [Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition](https://mlsanity.com/paper/2008.09412/) (2020)
- [POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery](https://mlsanity.com/paper/2303.13357/) (2023)

## Built on by (most cited first, 7 of 7)

- [Survey on Hand Gesture Recognition from Visual Input](https://mlsanity.com/paper/2501.11992/) (2025)
- [MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition](https://mlsanity.com/paper/2409.03890/) (2024)
- [ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognition](https://mlsanity.com/paper/2411.07118/) (2024)
- [Multi-task Learning For Joint Action and Gesture Recognition](https://mlsanity.com/paper/2505.17867/) (2025)
- [Advanced Acceptance Score: A Holistic Measure for Biometric Quantification](https://mlsanity.com/paper/2602.15535/) (2026)
- [UniCon-Former: Unified Convolution Transformer is All You Need for Hand Gesture Recognition](https://mlsanity.com/paper/2608.13217/) (2026)
- [Real-Time Hand Gesture Recognition for OpenXR Using Transformer-Based Machine Learning](https://mlsanity.com/paper/2609.25466/) (2026)

## Abstract (arXiv)

Transformer model have achieved state-of-the-art results in many applications like NLP, classification, etc. But their exploration in gesture recognition task is still limited. So, we propose a novel GestFormer architecture for dynamic hand gesture recognition. The motivation behind this design is to propose a resource efficient transformer model, since transformers are computationally expensive and very complex. So, we propose to use a pooling based token mixer named PoolFormer, since it uses only pooling layer which is a non-parametric layer instead of quadratic attention. The proposed model also leverages the space-invariant features of the wavelet transform and also the multiscale features are selected using multi-scale pooling. Further, a gated mechanism helps to focus on fine details of the gesture with the contextual information. This enhances the performance of the proposed model compared to the traditional transformer with fewer parameters, when evaluated on dynamic hand gesture datasets, NVidia Dynamic Hand Gesture and Briareo datasets. To prove the efficacy of the proposed model, we have experimented on single as well multimodal inputs such as infrared, normals, depth, optical flow and color images. We have also compared the proposed GestFormer in terms of resource efficiency and number of operations. The source code is available at https://github.com/mallikagarg/GestFormer.