# Multi-Scale Local-Temporal Similarity Fusion for Continuous Sign Language Recognition

arXiv: https://arxiv.org/abs/2107.12762 · HTML: https://mlsanity.com/paper/2107.12762/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Pixel-Adaptive Convolutional Neural Networks](https://mlsanity.com/paper/1904.05373/) (2019)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [Fully Convolutional Networks for Continuous Sign Language Recognition](https://mlsanity.com/paper/2007.12402/) (2020)
- [Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition](https://mlsanity.com/paper/2107.00606/) (2021)
- [SF-Net: Structured Feature Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/1908.01341/) (2019)

## Built on by (most cited first, 7 of 7)

- [Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition](https://mlsanity.com/paper/2506.03615/) (2025)
- [Multi-scale temporal network for continuous sign language recognition](https://mlsanity.com/paper/2204.03864/) (2022)
- [Continuous Sign Language Recognition Based on Motor attention mechanism and frame-level Self-distillation](https://mlsanity.com/paper/2402.19118/) (2024)
- [A Chinese Continuous Sign Language Dataset Based on Complex Environments](https://mlsanity.com/paper/2409.11960/) (2024)
- [Temporal superimposed crossover module for effective continuous sign language](https://mlsanity.com/paper/2211.03387/) (2022)
- [A Comparative Study of Continuous Sign Language Recognition Techniques](https://mlsanity.com/paper/2406.12369/) (2024)
- [USTM: Unified Spatial and Temporal Modeling for Continuous Sign Language Recognition](https://mlsanity.com/paper/2512.13415/) (2025)

## Abstract (arXiv)

Continuous sign language recognition (cSLR) is a public significant task that transcribes a sign language video into an ordered gloss sequence. It is important to capture the fine-grained gloss-level details, since there is no explicit alignment between sign video frames and the corresponding glosses. Among the past works, one promising way is to adopt a one-dimensional convolutional network (1D-CNN) to temporally fuse the sequential frames. However, CNNs are agnostic to similarity or dissimilarity, and thus are unable to capture local consistent semantics within temporally neighboring frames. To address the issue, we propose to adaptively fuse local features via temporal similarity for this task. Specifically, we devise a Multi-scale Local-Temporal Similarity Fusion Network (mLTSF-Net) as follows: 1) In terms of a specific video frame, we firstly select its similar neighbours with multi-scale receptive regions to accommodate different lengths of glosses. 2) To ensure temporal consistency, we then use position-aware convolution to temporally convolve each scale of selected frames. 3) To obtain a local-temporally enhanced frame-wise representation, we finally fuse the results of different scales using a content-dependent aggregator. We train our model in an end-to-end fashion, and the experimental results on RWTH-PHOENIX-Weather 2014 datasets (RWTH) demonstrate that our model achieves competitive performance compared with several state-of-the-art models.