Denoising-Contrastive Alignment for Continuous Sign Language Recognition
Not on the map. Why?
Abstract (arXiv)
Continuous sign language recognition (CSLR) aims to recognize signs in untrimmed sign language videos to textual glosses. A key challenge of CSLR is achieving effective cross-modality alignment between video and gloss sequences to enhance video representation. However, current cross-modality alignment paradigms often neglect the role of textual grammar to guide the video representation in learning global temporal context, which adversely affects recognition performance. To tackle this limitation, we propose a Denoising-Contrastive Alignment (DCA) paradigm. DCA creatively leverages textual grammar to enhance video representations through two complementary approaches: modeling the instance correspondence between signs and glosses from a discrimination perspective and aligning their global context from a generative perspective. Specifically, DCA accomplishes flexible instance-level correspondence between signs and glosses using a contrastive loss. Building on this, DCA models global context alignment between the video and gloss sequences by denoising the gloss representation from noise, guided by video representation. Additionally, DCA introduces gradient modulation to optimize the alignment and recognition gradients, ensuring a more effective learning process. By integrating gloss-wise and global context knowledge, DCA significantly enhances video representations for CSLR tasks. Experimental results across public benchmarks validate the effectiveness of DCA and confirm its video representation enhancement feasibility.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 49 references that are in the map. First posted on arXiv 5 May 2023.
Builds on
26 of 49 references.
References · 26
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2016Deep Transfer Learning with Joint Adaptation Networkscited by 612
- 2018Image Generation from Scene Graphscited by 324
14 more
- 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
- 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 179
- 2017Encoding Sentences with Graph Convolutional Networks for Semantic Role Labelingcited by 137
- 2022Two-Stream Network for Sign Language Recognition and Translationcited by 123
- 2021Visual Alignment Constraint for Continuous Sign Language Recognitioncited by 80
- 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
- 2023Continuous Sign Language Recognition with Correlation Networkcited by 56
- 2023DiffusionRet: Generative Text-Video Retrieval with Diffusion Modelcited by 52
- 2022MSDN: Mutually Semantic Distillation Network for Zero-Shot Learningcited by 48
- 2023CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignmentcited by 45
- 2020Boosting Continuous Sign Language Recognition via Cross Modality Augmentationcited by 40
- 2022Self-Emphasizing Network for Continuous Sign Language Recognitioncited by 33
- 2021PAC-Bayes Information Bottleneckcited by 28
- 2022Temporal Lift Pooling for Continuous Sign Language Recognitioncited by 26
Built on by
Not cited yet.