# USTM: Unified Spatial and Temporal Modeling for Continuous Sign Language Recognition

arXiv: https://arxiv.org/abs/2512.13415 · HTML: https://mlsanity.com/paper/2512.13415/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 30 of 36)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [Eigen-CAM: Class Activation Map using Principal Components](https://mlsanity.com/paper/2008.00299/) (2020)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [Fully Convolutional Networks for Continuous Sign Language Recognition](https://mlsanity.com/paper/2007.12402/) (2020)
- [Continuous Sign Language Recognition with Correlation Network](https://mlsanity.com/paper/2303.03202/) (2023)
- [CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment](https://mlsanity.com/paper/2303.05725/) (2023)
- [Boosting Continuous Sign Language Recognition via Cross Modality Augmentation](https://mlsanity.com/paper/2010.05264/) (2020)
- [Self-Emphasizing Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2211.17081/) (2022)
- [Temporal Lift Pooling for Continuous Sign Language Recognition](https://mlsanity.com/paper/2207.08734/) (2022)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [Self-Sufficient Framework for Continuous Sign Language Recognition](https://mlsanity.com/paper/2303.11771/) (2023)
- [AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition](https://mlsanity.com/paper/2308.08327/) (2023)
- [SlowFast Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2309.12304/) (2023)
- [Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2405.05672/) (2024)
- [Multi-Scale Local-Temporal Similarity Fusion for Continuous Sign Language Recognition](https://mlsanity.com/paper/2107.12762/) (2021)
- [Multi-View Spatial-Temporal Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2204.08747/) (2022)

## Built on by (most cited first, 1 of 1)

- [ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2607.03657/) (2026)

## Abstract (arXiv)

Continuous sign language recognition (CSLR) requires precise spatio-temporal modeling to accurately recognize sequences of gestures in videos. Existing frameworks often rely on CNN-based spatial backbones combined with temporal convolution or recurrent modules. These techniques fail in capturing fine-grained hand and facial cues and modeling long-range temporal dependencies. To address these limitations, we propose the Unified Spatio-Temporal Modeling (USTM) framework, a spatio-temporal encoder that effectively models complex patterns using a combination of a Swin Transformer backbone enhanced with lightweight temporal adapter with positional embeddings (TAPE). Our framework captures fine-grained spatial features alongside short and long-term temporal context, enabling robust sign language recognition from RGB videos without relying on multi-stream inputs or auxiliary modalities. Extensive experiments on benchmarked datasets including PHOENIX14, PHOENIX14T, and CSL-Daily demonstrate that USTM achieves state-of-the-art performance against RGB-based as well as multi-modal CSLR approaches, while maintaining competitive performance against multi-stream approaches. These results highlight the strength and efficacy of the USTM framework for CSLR. The code is available at https://github.com/gufranSabri/USTM