# Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition

arXiv: https://arxiv.org/abs/2102.05348 · HTML: https://mlsanity.com/paper/2102.05348/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [Multi-Context Attention for Human Pose Estimation](https://mlsanity.com/paper/1702.07432/) (2017)
- [Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition](https://mlsanity.com/paper/2008.09412/) (2020)
- [Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks](https://mlsanity.com/paper/1901.10323/) (2019)
- [Rank Pooling for Action Recognition](https://mlsanity.com/paper/1512.01848/) (2015)
- [Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training](https://mlsanity.com/paper/1812.06145/) (2018)
- [AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification](https://mlsanity.com/paper/2007.12034/) (2020)
- [A robust and efficient video representation for action recognition](https://mlsanity.com/paper/1504.05524/) (2015)
- [Zoom Out-and-In Network with Map Attention Decision for Region Proposal and Object Detection](https://mlsanity.com/paper/1709.04347/) (2017)
- [Cooperative Training of Deep Aggregation Networks for RGB-D Action Recognition](https://mlsanity.com/paper/1801.01080/) (2017)
- [Video Action Recognition Via Neural Architecture Searching](https://mlsanity.com/paper/1907.04632/) (2019)
- [Large-scale Isolated Gesture Recognition Using Convolutional Neural Networks](https://mlsanity.com/paper/1701.01814/) (2017)
- [Progression Modelling for Online and Early Gesture Detection](https://mlsanity.com/paper/1909.06672/) (2019)
- [SAR-NAS: Skeleton-based Action Recognition via Neural Architecture Searching](https://mlsanity.com/paper/2010.15336/) (2020)
- [Large-scale Continuous Gesture Recognition Using Convolutional Neural Networks](https://mlsanity.com/paper/1608.06338/) (2016)

## Built on by (most cited first, 10 of 10)

- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [Nested Collaborative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2203.15359/) (2022)
- [Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition](https://mlsanity.com/paper/2112.09129/) (2021)
- [A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition](https://mlsanity.com/paper/2211.09146/) (2022)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Tab2vox: CNN-Based Multivariate Multilevel Demand Forecasting Framework by Tabular-To-Voxel Image Conversion](https://mlsanity.com/paper/2209.10516/) (2022)
- [Multi-stage Factorized Spatio-Temporal Representation for RGB-D Action and Gesture Recognition](https://mlsanity.com/paper/2308.12006/) (2023)
- [Egocentric RGB+Depth Action Recognition in Industry-Like Settings](https://mlsanity.com/paper/2309.13962/) (2023)
- [Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation](https://mlsanity.com/paper/2405.08969/) (2024)
- [Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperation](https://mlsanity.com/paper/2109.08263/) (2021)

## Abstract (arXiv)

Human gesture recognition has drawn much attention in the area of computer vision. However, the performance of gesture recognition is always influenced by some gesture-irrelevant factors like the background and the clothes of performers. Therefore, focusing on the regions of hand/arm is important to the gesture recognition. Meanwhile, a more adaptive architecture-searched network structure can also perform better than the block-fixed ones like Resnet since it increases the diversity of features in different stages of the network better. In this paper, we propose a regional attention with architecture-rebuilt 3D network (RAAR3DNet) for gesture recognition. We replace the fixed Inception modules with the automatically rebuilt structure through the network via Neural Architecture Search (NAS), owing to the different shape and representation ability of features in the early, middle, and late stage of the network. It enables the network to capture different levels of feature representations at different layers more adaptively. Meanwhile, we also design a stackable regional attention module called dynamic-static Attention (DSA), which derives a Gaussian guidance heatmap and dynamic motion map to highlight the hand/arm regions and the motion information in the spatial and temporal domains, respectively. Extensive experiments on two recent large-scale RGB-D gesture datasets validate the effectiveness of the proposed method and show it outperforms state-of-the-art methods. The codes of our method are available at: https://github.com/zhoubenjia/RAAR3DNet.