# A Transformer-Based Multi-Stream Approach for Isolated Iranian Sign Language Recognition

arXiv: https://arxiv.org/abs/2407.09544 · HTML: https://mlsanity.com/paper/2407.09544/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [Spatial-Temporal Graph Convolutional Networks for Sign Language Recognition](https://mlsanity.com/paper/1901.11164/) (2019)
- [AI Fairness for People with Disabilities: Point of View](https://mlsanity.com/paper/1811.10670/) (2018)
- [DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation](https://mlsanity.com/paper/1802.07584/) (2018)
- [Score-level Multi Cue Fusion for Sign Language Recognition](https://mlsanity.com/paper/2009.14139/) (2020)
- [ChaLearn Looking at People: IsoGD and ConGD Large-scale RGB-D Gesture Recognition](https://mlsanity.com/paper/1907.12193/) (2019)
- [Continuous Sign Language Recognition Using Intra-inter Gloss Attention](https://mlsanity.com/paper/2406.18333/) (2024)
- [Word separation in continuous sign language using isolated signs and post-processing](https://mlsanity.com/paper/2204.00923/) (2022)

## Built on by (most cited first, 1 of 1)

- [TSLFormer: A Lightweight Transformer Model for Turkish Sign Language Recognition Using Skeletal Landmarks](https://mlsanity.com/paper/2505.07890/) (2025)

## Abstract (arXiv)

Sign language is an essential means of communication for millions of people around the world and serves as their primary language. However, most communication tools are developed for spoken and written languages which can cause problems and difficulties for the deaf and hard of hearing community. By developing a sign language recognition system, we can bridge this communication gap and enable people who use sign language as their main form of expression to better communicate with people and their surroundings. This recognition system increases the quality of health services, improves public services, and creates equal opportunities for the deaf community. This research aims to recognize Iranian Sign Language words with the help of the latest deep learning tools such as transformers. The dataset used includes 101 Iranian Sign Language words frequently used in academic environments such as universities. The network used is a combination of early fusion and late fusion transformer encoder-based networks optimized with the help of genetic algorithm. The selected features to train this network include hands and lips key points, and the distance and angle between hands extracted from the sign videos. Also, in addition to the training model for the classes, the embedding vectors of words are used as multi-task learning to have smoother and more efficient training. This model was also tested on sentences generated from our word dataset using a windowing technique for sentence translation. Finally, the sign language training software that provides real-time feedback to users with the help of the developed model, which has 90.2% accuracy on test data, was introduced, and in a survey, the effectiveness and efficiency of this type of sign language learning software and the impact of feedback were investigated.