# Visual Alignment Constraint for Continuous Sign Language Recognition

arXiv: https://arxiv.org/abs/2104.02330 · HTML: https://mlsanity.com/paper/2104.02330/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax](https://mlsanity.com/paper/2006.10408/) (2020)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective](https://mlsanity.com/paper/1908.08597/) (2019)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)
- [Transferring Cross-domain Knowledge for Video Sign Language Recognition](https://mlsanity.com/paper/2003.03703/) (2020)
- [Fully Convolutional Networks for Continuous Sign Language Recognition](https://mlsanity.com/paper/2007.12402/) (2020)
- [Z-Forcing: Training Stochastic Recurrent Networks](https://mlsanity.com/paper/1711.05411/) (2017)
- [Boosting Continuous Sign Language Recognition via Cross Modality Augmentation](https://mlsanity.com/paper/2010.05264/) (2020)
- [Hierarchical Multi Task Learning With CTC](https://mlsanity.com/paper/1807.07104/) (2018)
- [SF-Net: Structured Feature Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/1908.01341/) (2019)

## Built on by (most cited first, 30 of 80)

- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Continuous Sign Language Recognition with Correlation Network](https://mlsanity.com/paper/2303.03202/) (2023)
- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [Gloss Attention for Gloss-free Sign Language Translation](https://mlsanity.com/paper/2307.07361/) (2023)
- [Natural Language-Assisted Sign Language Recognition](https://mlsanity.com/paper/2303.12080/) (2023)
- [CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment](https://mlsanity.com/paper/2303.05725/) (2023)
- [Sign Language Translation from Instructional Videos](https://mlsanity.com/paper/2304.06371/) (2023)
- [Self-Emphasizing Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2211.17081/) (2022)
- [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024)
- [Improving Continuous Sign Language Recognition with Cross-Lingual Signs](https://mlsanity.com/paper/2308.10809/) (2023)
- [Temporal Lift Pooling for Continuous Sign Language Recognition](https://mlsanity.com/paper/2207.08734/) (2022)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [Sign Language Translation with Iterative Prototype](https://mlsanity.com/paper/2308.12191/) (2023)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Cross-modality Data Augmentation for End-to-End Sign Language Translation](https://mlsanity.com/paper/2305.11096/) (2023)
- [Towards Online Continuous Sign Language Recognition and Translation](https://mlsanity.com/paper/2401.05336/) (2024)
- [Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal](https://mlsanity.com/paper/2212.13023/) (2022)
- [Self-Sufficient Framework for Continuous Sign Language Recognition](https://mlsanity.com/paper/2303.11771/) (2023)
- [AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition](https://mlsanity.com/paper/2308.08327/) (2023)
- [SlowFast Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2309.12304/) (2023)
- [Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2405.05672/) (2024)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition](https://mlsanity.com/paper/2211.00448/) (2022)
- [Transcribing Natural Languages for The Deaf via Neural Editing Programs](https://mlsanity.com/paper/2112.09600/) (2021)
- [CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation](https://mlsanity.com/paper/2404.11111/) (2024)
- [EvSign: Sign Language Recognition and Translation with Streaming Events](https://mlsanity.com/paper/2407.12593/) (2024)
- [Deep Understanding of Sign Language for Sign to Subtitle Alignment](https://mlsanity.com/paper/2503.03287/) (2025)

## Abstract (arXiv)

Vision-based Continuous Sign Language Recognition (CSLR) aims to recognize unsegmented signs from image streams. Overfitting is one of the most critical problems in CSLR training, and previous works show that the iterative training scheme can partially solve this problem while also costing more training time. In this study, we revisit the iterative training scheme in recent CSLR works and realize that sufficient training of the feature extractor is critical to solving the overfitting problem. Therefore, we propose a Visual Alignment Constraint (VAC) to enhance the feature extractor with alignment supervision. Specifically, the proposed VAC comprises two auxiliary losses: one focuses on visual features only, and the other enforces prediction alignment between the feature extractor and the alignment module. Moreover, we propose two metrics to reflect overfitting by measuring the prediction inconsistency between the feature extractor and the alignment module. Experimental results on two challenging CSLR datasets show that the proposed VAC makes CSLR networks end-to-end trainable and achieves competitive performance.