# ConvMixFormer- A Resource-efficient Convolution Mixer for Transformer-based Dynamic Hand Gesture Recognition

arXiv: https://arxiv.org/abs/2411.07118 · HTML: https://mlsanity.com/paper/2411.07118/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [Patches Are All You Need?](https://mlsanity.com/paper/2201.09792/) (2022)
- [HRFormer: High-Resolution Transformer for Dense Prediction](https://mlsanity.com/paper/2110.09408/) (2021)
- [PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2303.17472/) (2023)
- [S2-MLP: Spatial-Shift MLP Architecture for Vision](https://mlsanity.com/paper/2106.07477/) (2021)
- [AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation](https://mlsanity.com/paper/2304.12301/) (2023)
- [Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition](https://mlsanity.com/paper/2008.09412/) (2020)
- [POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery](https://mlsanity.com/paper/2303.13357/) (2023)
- [Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks](https://mlsanity.com/paper/1901.10323/) (2019)
- [Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training](https://mlsanity.com/paper/1812.06145/) (2018)
- [A robust and efficient video representation for action recognition](https://mlsanity.com/paper/1504.05524/) (2015)
- [SepViT: Separable Vision Transformer](https://mlsanity.com/paper/2203.15380/) (2022)
- [Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition](https://mlsanity.com/paper/1804.07187/) (2018)
- [Res3ATN -- Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos](https://mlsanity.com/paper/2001.01083/) (2020)

## Built on by (most cited first, 2 of 2)

- [Survey on Hand Gesture Recognition from Visual Input](https://mlsanity.com/paper/2501.11992/) (2025)
- [UniCon-Former: Unified Convolution Transformer is All You Need for Hand Gesture Recognition](https://mlsanity.com/paper/2608.13217/) (2026)

## Abstract (arXiv)

Transformer models have demonstrated remarkable success in many domains such as natural language processing (NLP) and computer vision. With the growing interest in transformer-based architectures, they are now utilized for gesture recognition. So, we also explore and devise a novel ConvMixFormer architecture for dynamic hand gestures. The transformers use quadratic scaling of the attention features with the sequential data, due to which these models are computationally complex and heavy. We have considered this drawback of the transformer and designed a resource-efficient model that replaces the self-attention in the transformer with the simple convolutional layer-based token mixer. The computational cost and the parameters used for the convolution-based mixer are comparatively less than the quadratic self-attention. Convolution-mixer helps the model capture the local spatial features that self-attention struggles to capture due to their sequential processing nature. Further, an efficient gate mechanism is employed instead of a conventional feed-forward network in the transformer to help the model control the flow of features within different stages of the proposed model. This design uses fewer learnable parameters which is nearly half the vanilla transformer that helps in fast and efficient training. The proposed method is evaluated on NVidia Dynamic Hand Gesture and Briareo datasets and our model has achieved state-of-the-art results on single and multimodal inputs. We have also shown the parameter efficiency of the proposed ConvMixFormer model compared to other methods. The source code is available at https://github.com/mallikagarg/ConvMixFormer.