# Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

arXiv: https://arxiv.org/abs/2002.03187 · HTML: https://mlsanity.com/paper/2002.03187/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2020.

## Built on by (most cited first, 30 of 74)

- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [Continuous Sign Language Recognition with Correlation Network](https://mlsanity.com/paper/2303.03202/) (2023)
- [Gloss Attention for Gloss-free Sign Language Translation](https://mlsanity.com/paper/2307.07361/) (2023)
- [Natural Language-Assisted Sign Language Recognition](https://mlsanity.com/paper/2303.12080/) (2023)
- [CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment](https://mlsanity.com/paper/2303.05725/) (2023)
- [Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation](https://mlsanity.com/paper/2109.13318/) (2021)
- [Self-Emphasizing Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2211.17081/) (2022)
- [Content4All Open Research Sign Language Translation Datasets](https://mlsanity.com/paper/2105.02351/) (2021)
- [Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network](https://mlsanity.com/paper/2111.07258/) (2021)
- [Temporal Lift Pooling for Continuous Sign Language Recognition](https://mlsanity.com/paper/2207.08734/) (2022)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator](https://mlsanity.com/paper/2411.17799/) (2024)
- [Automatic dense annotation of large-vocabulary sign language videos](https://mlsanity.com/paper/2208.02802/) (2022)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Cross-modality Data Augmentation for End-to-End Sign Language Translation](https://mlsanity.com/paper/2305.11096/) (2023)
- [Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generation](https://mlsanity.com/paper/2207.03714/) (2022)
- [Towards Online Continuous Sign Language Recognition and Translation](https://mlsanity.com/paper/2401.05336/) (2024)
- [Sign Language Production with Latent Motion Transformer](https://mlsanity.com/paper/2312.12917/) (2023)
- [Scaling up sign spotting through sign language dictionaries](https://mlsanity.com/paper/2205.04152/) (2022)
- [PiSLTRc: Position-informed Sign Language Transformer with Content-aware Convolution](https://mlsanity.com/paper/2107.12600/) (2021)
- [Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal](https://mlsanity.com/paper/2212.13023/) (2022)
- [Self-Sufficient Framework for Continuous Sign Language Recognition](https://mlsanity.com/paper/2303.11771/) (2023)
- [A Token-level Contrastive Framework for Sign Language Translation](https://mlsanity.com/paper/2204.04916/) (2022)
- [AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition](https://mlsanity.com/paper/2308.08327/) (2023)
- [SlowFast Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2309.12304/) (2023)
- [Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization](https://mlsanity.com/paper/2506.15980/) (2025)
- [Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition](https://mlsanity.com/paper/2211.00448/) (2022)
- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [Fingerspelling Detection in American Sign Language](https://mlsanity.com/paper/2104.01291/) (2021)

## Abstract (arXiv)

Despite the recent success of deep learning in continuous sign language recognition (CSLR), deep models typically focus on the most discriminative features, ignoring other potentially non-trivial and informative contents. Such characteristic heavily constrains their capability to learn implicit visual grammars behind the collaboration of different visual cues (i,e., hand shape, facial expression and body posture). By injecting multi-cue learning into neural network design, we propose a spatial-temporal multi-cue (STMC) network to solve the vision-based sequence learning problem. Our STMC network consists of a spatial multi-cue (SMC) module and a temporal multi-cue (TMC) module. The SMC module is dedicated to spatial representation and explicitly decomposes visual features of different cues with the aid of a self-contained pose estimation branch. The TMC module models temporal correlations along two parallel paths, i.e., intra-cue and inter-cue, which aims to preserve the uniqueness and explore the collaboration of multiple cues. Finally, we design a joint optimization strategy to achieve the end-to-end sequence learning of the STMC network. To validate the effectiveness, we perform experiments on three large-scale CSLR benchmarks: PHOENIX-2014, CSL and PHOENIX-2014-T. Experimental results demonstrate that the proposed method achieves new state-of-the-art performance on all three benchmarks.