# TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions

arXiv: https://arxiv.org/abs/2403.11818 · HTML: https://mlsanity.com/paper/2403.11818/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [BiFormer: Vision Transformer with Bi-Level Routing Attention](https://mlsanity.com/paper/2303.08810/) (2023)
- [Neural GPUs Learn Algorithms](https://mlsanity.com/paper/1511.08228/) (2015)
- [TEINet: Towards an Efficient Architecture for Video Recognition](https://mlsanity.com/paper/1911.09435/) (2019)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)
- [Fully Convolutional Networks for Continuous Sign Language Recognition](https://mlsanity.com/paper/2007.12402/) (2020)
- [Continuous Sign Language Recognition with Correlation Network](https://mlsanity.com/paper/2303.03202/) (2023)
- [Quantitative Survey of the State of the Art in Sign Language Recognition](https://mlsanity.com/paper/2008.09918/) (2020)
- [Boosting Continuous Sign Language Recognition via Cross Modality Augmentation](https://mlsanity.com/paper/2010.05264/) (2020)
- [Discrete Autoencoders for Sequence Models](https://mlsanity.com/paper/1801.09797/) (2018)
- [SF-Net: Structured Feature Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/1908.01341/) (2019)
- [Can Active Memory Replace Attention?](https://mlsanity.com/paper/1610.08613/) (2016)

## Built on by (most cited first, 4 of 4)

- [A Chinese Continuous Sign Language Dataset Based on Complex Environments](https://mlsanity.com/paper/2409.11960/) (2024)
- [OLMD: Orientation-aware Long-term Motion Decoupling for Continuous Sign Language Recognition](https://mlsanity.com/paper/2503.08205/) (2025)
- [Hierarchical Sub-action Tree for Continuous Sign Language Recognition](https://mlsanity.com/paper/2506.20947/) (2025)
- [DESign: Dynamic Context-Aware Convolution and Efficient Subnet Regularization for Continuous Sign Language Recognition](https://mlsanity.com/paper/2507.03339/) (2025)

## Abstract (arXiv)

A key challenge in continuous sign language recognition (CSLR) is to efficiently capture long-range spatial interactions over time from the video input. To address this challenge, we propose TCNet, a hybrid network that effectively models spatio-temporal information from Trajectories and Correlated regions. TCNet's trajectory module transforms frames into aligned trajectories composed of continuous visual tokens. In addition, for a query token, self-attention is learned along the trajectory. As such, our network can also focus on fine-grained spatio-temporal patterns, such as finger movements, of a specific region in motion. TCNet's correlation module uses a novel dynamic attention mechanism that filters out irrelevant frame regions. Additionally, it assigns dynamic key-value tokens from correlated regions to each query. Both innovations significantly reduce the computation cost and memory. We perform experiments on four large-scale datasets: PHOENIX14, PHOENIX14-T, CSL, and CSL-Daily, respectively. Our results demonstrate that TCNet consistently achieves state-of-the-art performance. For example, we improve over the previous state-of-the-art by 1.5% and 1.0% word error rate on PHOENIX14 and PHOENIX14-T, respectively.