# OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

arXiv: https://arxiv.org/abs/2110.05877 · HTML: https://mlsanity.com/paper/2110.05877/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1703.08274/) (2017)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [A Comprehensive Study of Deep Video Action Recognition](https://mlsanity.com/paper/2012.06567/) (2020)
- [AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods](https://mlsanity.com/paper/2008.00932/) (2020)
- [BertGCN: Transductive Text Classification by Combining GCN and BERT](https://mlsanity.com/paper/2105.05727/) (2021)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Quantitative Survey of the State of the Art in Sign Language Recognition](https://mlsanity.com/paper/2008.09918/) (2020)
- [Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning](https://mlsanity.com/paper/1805.02335/) (2018)
- [Global-local Enhancement Network for NMFs-aware Sign Language Recognition](https://mlsanity.com/paper/2008.10428/) (2020)

## Built on by (most cited first, 30 of 31)

- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [Natural Language-Assisted Sign Language Recognition](https://mlsanity.com/paper/2303.12080/) (2023)
- [BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization](https://mlsanity.com/paper/2302.05075/) (2023)
- [ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition](https://mlsanity.com/paper/2304.05934/) (2023)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [SignCLIP: Connecting Text and Sign Language by Contrastive Learning](https://mlsanity.com/paper/2407.01264/) (2024)
- [Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas](https://mlsanity.com/paper/2403.02563/) (2024)
- [The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes](https://mlsanity.com/paper/2310.00196/) (2023)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition](https://mlsanity.com/paper/2212.12857/) (2022)
- [Guided Code Generation with LLMs: A Multi-Agent Framework for Complex Code Tasks](https://mlsanity.com/paper/2501.06625/) (2025)
- [Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition](https://mlsanity.com/paper/2406.10501/) (2024)
- [iSign: A Benchmark for Indian Sign Language Processing](https://mlsanity.com/paper/2407.05404/) (2024)
- [Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for Isolated Indian Sign Language Recognition](https://mlsanity.com/paper/2407.14224/) (2024)
- [Exploring Strategies for Modeling Sign Language Phonology](https://mlsanity.com/paper/2310.00195/) (2023)
- [Transfer Learning for Cross-dataset Isolated Sign Language Recognition in Under-Resourced Datasets](https://mlsanity.com/paper/2403.14534/) (2024)
- [MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production](https://mlsanity.com/paper/2407.12842/) (2024)
- [Improving Sign Recognition with Phonology](https://mlsanity.com/paper/2302.05759/) (2023)
- [Modelling the Distribution of Human Motion for Sign Language Assessment](https://mlsanity.com/paper/2408.10073/) (2024)
- [3D-LEX v1.0: 3D Lexicons for American Sign Language and Sign Language of the Netherlands](https://mlsanity.com/paper/2409.01901/) (2024)
- [ISLTranslate: Dataset for Translating Indian Sign Language](https://mlsanity.com/paper/2307.05440/) (2023)
- [The NGT200 Dataset: Geometric Multi-View Isolated Sign Recognition](https://mlsanity.com/paper/2409.15284/) (2024)
- [AirLetters: An Open Video Dataset of Characters Drawn in the Air](https://mlsanity.com/paper/2410.02921/) (2024)
- [Studying and Mitigating Biases in Sign Language Understanding Models](https://mlsanity.com/paper/2410.05206/) (2024)
- [AzSLD: Azerbaijani Sign Language Dataset for Fingerspelling, Word, and Sentence Translation with Baseline Software](https://mlsanity.com/paper/2411.12865/) (2024)
- [A Comparative Analysis of Techniques and Algorithms for Recognising Sign Language](https://mlsanity.com/paper/2305.13941/) (2023)
- [SSLR: A Semi-Supervised Learning Method for Isolated Sign Language Recognition](https://mlsanity.com/paper/2504.16640/) (2025)
- [Generation of Indian Sign Language Letters, Numbers, and Words](https://mlsanity.com/paper/2508.09522/) (2025)
- [HandCraft: Dynamic Sign Generation for Synthetic Data Augmentation](https://mlsanity.com/paper/2508.14345/) (2025)
- [Isolated Sign Language Recognition for Icelandic Sign Language: Experiments in a Low-resource Setting](https://mlsanity.com/paper/2609.25862/) (2026)

## Abstract (arXiv)

AI technologies for Natural Languages have made tremendous progress recently. However, commensurate progress has not been made on Sign Languages, in particular, in recognizing signs as individual words or as complete sentences. We introduce OpenHands, a library where we take four key ideas from the NLP community for low-resource languages and apply them to sign languages for word-level recognition. First, we propose using pose extracted through pretrained models as the standard modality of data to reduce training time and enable efficient inference, and we release standardized pose datasets for 6 different sign languages - American, Argentinian, Chinese, Greek, Indian, and Turkish. Second, we train and release checkpoints of 4 pose-based isolated sign language recognition models across all 6 languages, providing baselines and ready checkpoints for deployment. Third, to address the lack of labelled data, we propose self-supervised pretraining on unlabelled data. We curate and release the largest pose-based pretraining dataset on Indian Sign Language (Indian-SL). Fourth, we compare different pretraining strategies and for the first time establish that pretraining is effective for sign language recognition by demonstrating (a) improved fine-tuning performance especially in low-resource settings, and (b) high crosslingual transfer from Indian-SL to few other sign languages. We open-source all models and datasets in OpenHands with a hope that it makes research in sign languages more accessible, available here at https://github.com/AI4Bharat/OpenHands .