# Multi-Modal Zero-Shot Sign Language Recognition

arXiv: https://arxiv.org/abs/2109.00796 · HTML: https://mlsanity.com/paper/2109.00796/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Learning Attributes Equals Multi-Source Domain Generalization](https://mlsanity.com/paper/1605.00743/) (2016)
- [Towards Universal Representation for Unseen Action Recognition](https://mlsanity.com/paper/1803.08460/) (2018)
- [A Generative Approach to Zero-Shot and Few-Shot Action Recognition](https://mlsanity.com/paper/1801.09086/) (2018)
- [Zero-Shot Visual Recognition via Bidirectional Latent Embedding](https://mlsanity.com/paper/1607.02104/) (2016)
- [Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation](https://mlsanity.com/paper/1611.08663/) (2016)
- [Sign Language Production: A Review](https://mlsanity.com/paper/2103.15910/) (2021)
- [Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2101.11530/) (2021)
- [Semantic Embedding Space for Zero-Shot Action Recognition](https://mlsanity.com/paper/1502.01540/) (2015)
- [Action2Vec: A Crossmodal Embedding Approach to Action Learning](https://mlsanity.com/paper/1901.00484/) (2019)
- [Skeleton-based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module](https://mlsanity.com/paper/1811.07081/) (2018)
- [Zero-Shot Sign Language Recognition: Can Textual Data Uncover Sign Languages?](https://mlsanity.com/paper/1907.10292/) (2019)

## Built on by (most cited first, 3 of 3)

- [All You Need In Sign Language Production](https://mlsanity.com/paper/2201.01609/) (2022)
- [Word separation in continuous sign language using isolated signs and post-processing](https://mlsanity.com/paper/2204.00923/) (2022)
- [A Non-Anatomical Graph Structure for isolated hand gesture separation in continuous gesture sequences](https://mlsanity.com/paper/2207.07619/) (2022)

## Abstract (arXiv)

Zero-Shot Learning (ZSL) has rapidly advanced in recent years. Towards overcoming the annotation bottleneck in the Sign Language Recognition (SLR), we explore the idea of Zero-Shot Sign Language Recognition (ZS-SLR) with no annotated visual examples, by leveraging their textual descriptions. In this way, we propose a multi-modal Zero-Shot Sign Language Recognition (ZS-SLR) model harnessing from the complementary capabilities of deep features fused with the skeleton-based ones. A Transformer-based model along with a C3D model is used for hand detection and deep features extraction, respectively. To make a trade-off between the dimensionality of the skeletonbased and deep features, we use an Auto-Encoder (AE) on top of the Long Short Term Memory (LSTM) network. Finally, a semantic space is used to map the visual features to the lingual embedding of the class labels, achieved via the Bidirectional Encoder Representations from Transformers (BERT) model. Results on four large-scale datasets, RKS-PERSIANSIGN, First-Person, ASLVID, and isoGD, show the superiority of the proposed model compared to state-of-the-art alternatives in ZS-SLR.