# Using Motion History Images with 3D Convolutional Networks in Isolated Sign Language Recognition

arXiv: https://arxiv.org/abs/2110.12396 · HTML: https://mlsanity.com/paper/2110.12396/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods](https://mlsanity.com/paper/2008.00932/) (2020)
- [Quantitative Survey of the State of the Art in Sign Language Recognition](https://mlsanity.com/paper/2008.09918/) (2020)
- [Global-local Enhancement Network for NMFs-aware Sign Language Recognition](https://mlsanity.com/paper/2008.10428/) (2020)
- [Fingerspelling recognition in the wild with iterative visual attention](https://mlsanity.com/paper/1908.10546/) (2019)
- [Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognition](https://mlsanity.com/paper/2104.10166/) (2021)
- [BosphorusSign22k Sign Language Recognition Dataset](https://mlsanity.com/paper/2004.01283/) (2020)
- [Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video](https://mlsanity.com/paper/1506.01911/) (2015)
- [ChaLearn LAP Large Scale Signer Independent Isolated Sign Language Recognition Challenge: Design, Results and Future Research](https://mlsanity.com/paper/2105.05066/) (2021)
- [Dynamic Gesture Recognition by Using CNNs and Star RGB: a Temporal Information Condensation](https://mlsanity.com/paper/1904.08505/) (2019)
- [Score-level Multi Cue Fusion for Sign Language Recognition](https://mlsanity.com/paper/2009.14139/) (2020)

## Built on by (most cited first, 12 of 12)

- [Isolated Sign Language Recognition based on Tree Structure Skeleton Images](https://mlsanity.com/paper/2304.05403/) (2023)
- [Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse](https://mlsanity.com/paper/2308.09622/) (2023)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [Survey on Hand Gesture Recognition from Visual Input](https://mlsanity.com/paper/2501.11992/) (2025)
- [Transfer Learning for Cross-dataset Isolated Sign Language Recognition in Under-Resourced Datasets](https://mlsanity.com/paper/2403.14534/) (2024)
- [Continuous Sign Language Recognition via Temporal Super-Resolution Network](https://mlsanity.com/paper/2207.00928/) (2022)
- [E-TSL: A Continuous Educational Turkish Sign Language Dataset with Baseline Methods](https://mlsanity.com/paper/2405.02984/) (2024)
- [TSLFormer: A Lightweight Transformer Model for Turkish Sign Language Recognition Using Skeletal Landmarks](https://mlsanity.com/paper/2505.07890/) (2025)
- [Fine-Tuning Video Transformers for Word-Level Bangla Sign Language: A Comparative Analysis for Classification Tasks](https://mlsanity.com/paper/2506.04367/) (2025)
- [Advancements in Tactile Hand Gesture Recognition for Enhanced Human-Machine Interaction](https://mlsanity.com/paper/2405.17038/) (2024)
- [Sign language recognition based on deep learning and low-cost handcrafted descriptors](https://mlsanity.com/paper/2408.07244/) (2024)
- [ISLR101: an Iranian Word-Level Sign Language Recognition Dataset](https://mlsanity.com/paper/2503.12451/) (2025)

## Abstract (arXiv)

Sign language recognition using computational models is a challenging problem that requires simultaneous spatio-temporal modeling of the multiple sources, i.e. faces, hands, body, etc. In this paper, we propose an isolated sign language recognition model based on a model trained using Motion History Images (MHI) that are generated from RGB video frames. RGB-MHI images represent spatio-temporal summary of each sign video effectively in a single RGB image. We propose two different approaches using this RGB-MHI model. In the first approach, we use the RGB-MHI model as a motion-based spatial attention module integrated into a 3D-CNN architecture. In the second approach, we use RGB-MHI model features directly with the features of a 3D-CNN model using a late fusion technique. We perform extensive experiments on two recently released large-scale isolated sign language datasets, namely AUTSL and BosphorusSign22k. Our experiments show that our models, which use only RGB data, can compete with the state-of-the-art models in the literature that use multi-modal data.