# Continuous Sign Language Recognition with Correlation Network

arXiv: https://arxiv.org/abs/2303.03202 · HTML: https://mlsanity.com/paper/2303.03202/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 60 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Convolutional neural network architecture for geometric matching](https://mlsanity.com/paper/1703.05593/) (2017)
- [Detect to Track and Track to Detect](https://mlsanity.com/paper/1710.03958/) (2017)
- [TEINet: Towards an Efficient Architecture for Video Recognition](https://mlsanity.com/paper/1911.09435/) (2019)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)
- [Video Modeling with Correlation Networks](https://mlsanity.com/paper/1906.03349/) (2019)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://mlsanity.com/paper/2110.05382/) (2021)
- [Fully Convolutional Networks for Continuous Sign Language Recognition](https://mlsanity.com/paper/2007.12402/) (2020)
- [Boosting Continuous Sign Language Recognition via Cross Modality Augmentation](https://mlsanity.com/paper/2010.05264/) (2020)
- [Spatio-Temporal Channel Correlation Networks for Action Classification](https://mlsanity.com/paper/1806.07754/) (2018)
- [Pose-based Sign Language Recognition using GCN and BERT](https://mlsanity.com/paper/2012.00781/) (2020)
- [Motion Feature Network: Fixed Motion Filter for Action Recognition](https://mlsanity.com/paper/1807.10037/) (2018)

## Built on by (most cited first, 30 of 56)

- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Deep Correlated Prompting for Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2410.06558/) (2024)
- [Towards Online Continuous Sign Language Recognition and Translation](https://mlsanity.com/paper/2401.05336/) (2024)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation](https://mlsanity.com/paper/2506.00129/) (2025)
- [SlowFast Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2309.12304/) (2023)
- [Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2405.05672/) (2024)
- [An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMs](https://mlsanity.com/paper/2408.10593/) (2024)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Pose-Guided Fine-Grained Sign Language Video Generation](https://mlsanity.com/paper/2409.16709/) (2024)
- [CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation](https://mlsanity.com/paper/2404.11111/) (2024)
- [LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression](https://mlsanity.com/paper/2509.00419/) (2025)
- [EvSign: Sign Language Recognition and Translation with Streaming Events](https://mlsanity.com/paper/2407.12593/) (2024)
- [Deep Understanding of Sign Language for Sign to Subtitle Alignment](https://mlsanity.com/paper/2503.03287/) (2025)
- [Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition](https://mlsanity.com/paper/2506.03615/) (2025)
- [SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning](https://mlsanity.com/paper/2401.11847/) (2024)
- [Continuous Sign Language Recognition Using Intra-inter Gloss Attention](https://mlsanity.com/paper/2406.18333/) (2024)
- [SCOPE: Sign Language Contextual Processing with Embedding from LLMs](https://mlsanity.com/paper/2409.01073/) (2024)
- [TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions](https://mlsanity.com/paper/2403.11818/) (2024)
- [CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognition](https://mlsanity.com/paper/2504.01666/) (2025)
- [SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Space](https://mlsanity.com/paper/2504.16315/) (2025)
- [Continuous Sign Language Recognition Based on Motor attention mechanism and frame-level Self-distillation](https://mlsanity.com/paper/2402.19118/) (2024)
- [Text-Driven Diffusion Model for Sign Language Production](https://mlsanity.com/paper/2503.15914/) (2025)
- [A Chinese Continuous Sign Language Dataset Based on Complex Environments](https://mlsanity.com/paper/2409.11960/) (2024)
- [Stable Signer: Hierarchical Sign Language Generative Model](https://mlsanity.com/paper/2512.04048/) (2025)
- [A Comparative Study of Continuous Sign Language Recognition Techniques](https://mlsanity.com/paper/2406.12369/) (2024)
- [Training Strategies for Isolated Sign Language Recognition](https://mlsanity.com/paper/2412.11553/) (2024)

## Abstract (arXiv)

Human body trajectories are a salient cue to identify actions in the video. Such body trajectories are mainly conveyed by hands and face across consecutive frames in sign language. However, current methods in continuous sign language recognition (CSLR) usually process frames independently, thus failing to capture cross-frame trajectories to effectively identify a sign. To handle this limitation, we propose correlation network (CorrNet) to explicitly capture and leverage body trajectories across frames to identify signs. In specific, a correlation module is first proposed to dynamically compute correlation maps between the current frame and adjacent frames to identify trajectories of all spatial patches. An identification module is then presented to dynamically emphasize the body trajectories within these correlation maps. As a result, the generated features are able to gain an overview of local temporal movements to identify a sign. Thanks to its special attention on body trajectories, CorrNet achieves new state-of-the-art accuracy on four large-scale datasets, i.e., PHOENIX14, PHOENIX14-T, CSL-Daily, and CSL. A comprehensive comparison with previous spatial-temporal reasoning methods verifies the effectiveness of CorrNet. Visualizations demonstrate the effects of CorrNet on emphasizing human body trajectories across adjacent frames.