# Sign Language Recognition Analysis using Multimodal Data

arXiv: https://arxiv.org/abs/1909.11232 · HTML: https://mlsanity.com/paper/1909.11232/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2019.

## Built on by (most cited first, 4 of 4)

- [Fingerspelling PoseNet: Enhancing Fingerspelling Translation with Pose-Based Transformer Models](https://mlsanity.com/paper/2311.12128/) (2023)
- [Sign Language Recognition using Bidirectional Reservoir Computing](https://mlsanity.com/paper/2512.00777/) (2025)
- [Sign Language Recognition using Parallel Bidirectional Reservoir Computing](https://mlsanity.com/paper/2512.19451/) (2025)
- [A Low-Cost Hybrid Reservoir Computing Model for Isolated Sign Language Video Recognition](https://mlsanity.com/paper/2608.03444/) (2026)

## Abstract (arXiv)

Voice-controlled personal and home assistants (such as the Amazon Echo and Apple Siri) are becoming increasingly popular for a variety of applications. However, the benefits of these technologies are not readily accessible to Deaf or Hard-ofHearing (DHH) users. The objective of this study is to develop and evaluate a sign recognition system using multiple modalities that can be used by DHH signers to interact with voice-controlled devices. With the advancement of depth sensors, skeletal data is used for applications like video analysis and activity recognition. Despite having similarity with the well-studied human activity recognition, the use of 3D skeleton data in sign language recognition is rare. This is because unlike activity recognition, sign language is mostly dependent on hand shape pattern. In this work, we investigate the feasibility of using skeletal and RGB video data for sign language recognition using a combination of different deep learning architectures. We validate our results on a large-scale American Sign Language (ASL) dataset of 12 users and 13107 samples across 51 signs. It is named as GMUASL51. We collected the dataset over 6 months and it will be publicly released in the hope of spurring further machine learning research towards providing improved accessibility for digital assistants.