# Word-level Sign Language Recognition with Multi-stream Neural Networks Focusing on Local Regions and Skeletal Information

arXiv: https://arxiv.org/abs/2106.15989 · HTML: https://mlsanity.com/paper/2106.15989/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 72 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 26 of 26)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://mlsanity.com/paper/1811.00751/) (2018)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition](https://mlsanity.com/paper/2010.09978/) (2020)
- [Transferring Cross-domain Knowledge for Video Sign Language Recognition](https://mlsanity.com/paper/2003.03703/) (2020)
- [Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks](https://mlsanity.com/paper/1809.03355/) (2018)
- [Pose-based Sign Language Recognition using GCN and BERT](https://mlsanity.com/paper/2012.00781/) (2020)
- [Spatial-Temporal Graph Convolutional Networks for Sign Language Recognition](https://mlsanity.com/paper/1901.11164/) (2019)
- [Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video](https://mlsanity.com/paper/1506.01911/) (2015)
- [Attention-based Temporal Weighted Convolutional Neural Network for Action Recognition](https://mlsanity.com/paper/1803.07179/) (2018)

## Built on by (most cited first, 3 of 3)

- [BdSLW60: A Word-Level Bangla Sign Language Dataset](https://mlsanity.com/paper/2402.08635/) (2024)
- [SplashNet: Split-and-Share Encoders for Accurate and Efficient Typing with Surface Electromyography](https://mlsanity.com/paper/2506.12356/) (2025)
- [IsoSignVid2Aud: Sign Language Video to Audio Conversion without Text Intermediaries](https://mlsanity.com/paper/2510.07837/) (2025)

## Abstract (arXiv)

Word-level sign language recognition (WSLR) has attracted attention because it is expected to overcome the communication barrier between people with speech impairment and those who can hear. In the WSLR problem, a method designed for action recognition has achieved the state-of-the-art accuracy. Indeed, it sounds reasonable for an action recognition method to perform well on WSLR because sign language is regarded as an action. However, a careful evaluation of the tasks reveals that the tasks of action recognition and WSLR are inherently different. Hence, in this paper, we propose a novel WSLR method that takes into account information specifically useful for the WSLR problem. We realize it as a multi-stream neural network (MSNN), which consist of three streams: 1) base stream, 2) local image stream, and 3) skeleton stream. Each stream is designed to handle different types of information. The base stream deals with quick and detailed movements of the hands and body, the local image stream focuses on handshapes and facial expressions, and the skeleton stream captures the relative positions of the body and both hands. This approach allows us to combine various types of data for more comprehensive gesture analysis. Experimental results on the WLASL and MS-ASL datasets show the effectiveness of the proposed method; it achieved an improvement of approximately 10%--15% in Top-1 accuracy when compared with conventional methods.