# ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos

arXiv: https://arxiv.org/abs/2108.10059 · HTML: https://mlsanity.com/paper/2108.10059/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 52 references that are in the map.
- First posted on arXiv 23 Aug 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Learning Robust Visual-Semantic Embeddings](https://mlsanity.com/paper/1703.05908/) (2017)
- [Learning Attributes Equals Multi-Source Domain Generalization](https://mlsanity.com/paper/1605.00743/) (2016)
- [Towards Universal Representation for Unseen Action Recognition](https://mlsanity.com/paper/1803.08460/) (2018)
- [A Generative Approach to Zero-Shot and Few-Shot Action Recognition](https://mlsanity.com/paper/1801.09086/) (2018)
- [Zero-Shot Visual Recognition via Bidirectional Latent Embedding](https://mlsanity.com/paper/1607.02104/) (2016)
- [Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation](https://mlsanity.com/paper/1611.08663/) (2016)
- [Sign Language Production: A Review](https://mlsanity.com/paper/2103.15910/) (2021)
- [Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2101.11530/) (2021)
- [Semantic Embedding Space for Zero-Shot Action Recognition](https://mlsanity.com/paper/1502.01540/) (2015)
- [Action2Vec: A Crossmodal Embedding Approach to Action Learning](https://mlsanity.com/paper/1901.00484/) (2019)
- [Skeleton-based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module](https://mlsanity.com/paper/1811.07081/) (2018)
- [Zero-Shot Sign Language Recognition: Can Textual Data Uncover Sign Languages?](https://mlsanity.com/paper/1907.10292/) (2019)

## Built on by (most cited first, 4 of 4)

- [All You Need In Sign Language Production](https://mlsanity.com/paper/2201.01609/) (2022)
- [Word separation in continuous sign language using isolated signs and post-processing](https://mlsanity.com/paper/2204.00923/) (2022)
- [A Non-Anatomical Graph Structure for isolated hand gesture separation in continuous gesture sequences](https://mlsanity.com/paper/2207.07619/) (2022)
- [Representing Signs as Signs: One-Shot ISLR to Facilitate Functional Sign Language Technologies](https://mlsanity.com/paper/2502.20171/) (2025)

## Abstract (arXiv)

Sign Language Recognition (SLR) is a challenging research area in computer vision. To tackle the annotation bottleneck in SLR, we formulate the problem of Zero-Shot Sign Language Recognition (ZS-SLR) and propose a two-stream model from two input modalities: RGB and Depth videos. To benefit from the vision Transformer capabilities, we use two vision Transformer models, for human detection and visual features representation. We configure a transformer encoder-decoder architecture, as a fast and accurate human detection model, to overcome the challenges of the current human detection models. Considering the human keypoints, the detected human body is segmented into nine parts. A spatio-temporal representation from human body is obtained using a vision Transformer and a LSTM network. A semantic space maps the visual features to the lingual embedding of the class labels via a Bidirectional Encoder Representations from Transformers (BERT) model. We evaluated the proposed model on four datasets, Montalbano II, MSR Daily Activity 3D, CAD-60, and NTU-60, obtaining state-of-the-art results compared to state-of-the-art ZS-SLR models.