# Transferring Cross-domain Knowledge for Video Sign Language Recognition

arXiv: https://arxiv.org/abs/2003.03703 · HTML: https://mlsanity.com/paper/2003.03703/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2020.

## Built on by (most cited first, 30 of 60)

- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://mlsanity.com/paper/2110.05382/) (2021)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring](https://mlsanity.com/paper/2103.04260/) (2021)
- [Gloss Attention for Gloss-free Sign Language Translation](https://mlsanity.com/paper/2307.07361/) (2023)
- [Natural Language-Assisted Sign Language Recognition](https://mlsanity.com/paper/2303.12080/) (2023)
- [BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization](https://mlsanity.com/paper/2302.05075/) (2023)
- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2303.12793/) (2023)
- [Improving Continuous Sign Language Recognition with Cross-Lingual Signs](https://mlsanity.com/paper/2308.10809/) (2023)
- [Aligning Subtitles in Sign Language Videos](https://mlsanity.com/paper/2105.02877/) (2021)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval](https://mlsanity.com/paper/2408.09949/) (2024)
- [Sign Language Video Retrieval with Free-Form Textual Queries](https://mlsanity.com/paper/2201.02495/) (2022)
- [Fusing Higher-order Features in Graph Neural Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2105.01563/) (2021)
- [Automatic dense annotation of large-vocabulary sign language videos](https://mlsanity.com/paper/2208.02802/) (2022)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Beyond Monocular Deraining: Parallel Stereo Deraining Network Via Semantic Prior](https://mlsanity.com/paper/2105.03830/) (2021)
- [Isolated Sign Language Recognition based on Tree Structure Skeleton Images](https://mlsanity.com/paper/2304.05403/) (2023)
- [Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse](https://mlsanity.com/paper/2308.09622/) (2023)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [Scaling up sign spotting through sign language dictionaries](https://mlsanity.com/paper/2205.04152/) (2022)
- [Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2211.10412/) (2022)
- [Sign Segmentation with Changepoint-Modulated Pseudo-Labelling](https://mlsanity.com/paper/2104.13817/) (2021)
- [Weakly-supervised Fingerspelling Recognition in British Sign Language Videos](https://mlsanity.com/paper/2211.08954/) (2022)
- [StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition](https://mlsanity.com/paper/2212.12857/) (2022)

## Abstract (arXiv)

Word-level sign language recognition (WSLR) is a fundamental task in sign language interpretation. It requires models to recognize isolated sign words from videos. However, annotating WSLR data needs expert knowledge, thus limiting WSLR dataset acquisition. On the contrary, there are abundant subtitled sign news videos on the internet. Since these videos have no word-level annotation and exhibit a large domain gap from isolated signs, they cannot be directly used for training WSLR models. We observe that despite the existence of a large domain gap, isolated and news signs share the same visual concepts, such as hand gestures and body movements. Motivated by this observation, we propose a novel method that learns domain-invariant visual concepts and fertilizes WSLR models by transferring knowledge of subtitled news sign to them. To this end, we extract news signs using a base WSLR model, and then design a classifier jointly trained on news and isolated signs to coarsely align these two domain features. In order to learn domain-invariant features within each class and suppress domain-specific features, our method further resorts to an external memory to store the class centroids of the aligned news signs. We then design a temporal attention based on the learnt descriptor to improve recognition performance. Experimental results on standard WSLR datasets show that our method outperforms previous state-of-the-art methods significantly. We also demonstrate the effectiveness of our method on automatically localizing signs from sign news, achieving 28.1 for AP@0.5.