# Sign language recognition based on deep learning and low-cost handcrafted descriptors

arXiv: https://arxiv.org/abs/2408.07244 · HTML: https://mlsanity.com/paper/2408.07244/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Builds on (most cited first, 12 of 12)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods](https://mlsanity.com/paper/2008.00932/) (2020)
- [Using Motion History Images with 3D Convolutional Networks in Isolated Sign Language Recognition](https://mlsanity.com/paper/2110.12396/) (2021)
- [ChaLearn LAP Large Scale Signer Independent Isolated Sign Language Recognition Challenge: Design, Results and Future Research](https://mlsanity.com/paper/2105.05066/) (2021)
- [Score-level Multi Cue Fusion for Sign Language Recognition](https://mlsanity.com/paper/2009.14139/) (2020)
- [Independent Sign Language Recognition with 3D Body, Hands, and Face Reconstruction](https://mlsanity.com/paper/2012.05698/) (2020)

## Abstract (arXiv)

In recent years, deep learning techniques have been used to develop sign language recognition systems, potentially serving as a communication tool for millions of hearing-impaired individuals worldwide. However, there are inherent challenges in creating such systems. Firstly, it is important to consider as many linguistic parameters as possible in gesture execution to avoid ambiguity between words. Moreover, to facilitate the real-world adoption of the created solution, it is essential to ensure that the chosen technology is realistic, avoiding expensive, intrusive, or low-mobility sensors, as well as very complex deep learning architectures that impose high computational requirements. Based on this, our work aims to propose an efficient sign language recognition system that utilizes low-cost sensors and techniques. To this end, an object detection model was trained specifically for detecting the interpreter's face and hands, ensuring focus on the most relevant regions of the image and generating inputs with higher semantic value for the classifier. Additionally, we introduced a novel approach to obtain features representing hand location and movement by leveraging spatial information derived from centroid positions of bounding boxes, thereby enhancing sign discrimination. The results demonstrate the efficiency of our handcrafted features, increasing accuracy by 7.96% on the AUTSL dataset, while adding fewer than 700 thousand parameters and incurring less than 10 milliseconds of additional inference time. These findings highlight the potential of our technique to strike a favorable balance between computational cost and accuracy, making it a promising approach for practical sign language recognition applications.