# Improvement in Sign Language Translation Using Text CTC Alignment

arXiv: https://arxiv.org/abs/2412.09014 · HTML: https://mlsanity.com/paper/2412.09014/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 47 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 4 of 4)

- [Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions](https://mlsanity.com/paper/2104.02724/) (2021)
- [Data Augmentation for Sign Language Gloss Translation](https://mlsanity.com/paper/2105.07476/) (2021)
- [Hierarchical Multitask Learning for CTC-based Speech Recognition](https://mlsanity.com/paper/1807.06234/) (2018)
- [Self-Supervised Video Transformers for Isolated Sign Language Recognition](https://mlsanity.com/paper/2309.02450/) (2023)

## Built on by (most cited first, 5 of 5)

- [Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Model](https://mlsanity.com/paper/2505.24355/) (2025)
- [Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards](https://mlsanity.com/paper/2606.19352/) (2026)
- [Q-BridgeNet: A Quantization Network for Cross-Lingual Sign Language Translation](https://mlsanity.com/paper/2607.11215/) (2026)
- [ODE-Based Transformer Decoders for Iterative Sign Language Translation](https://mlsanity.com/paper/2608.11352/) (2026)
- [SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale](https://mlsanity.com/paper/2609.35225/) (2026)

## Abstract (arXiv)

Current sign language translation (SLT) approaches often rely on gloss-based supervision with Connectionist Temporal Classification (CTC), limiting their ability to handle non-monotonic alignments between sign language video and spoken text. In this work, we propose a novel method combining joint CTC/Attention and transfer learning. The joint CTC/Attention introduces hierarchical encoding and integrates CTC with the attention mechanism during decoding, effectively managing both monotonic and non-monotonic alignments. Meanwhile, transfer learning helps bridge the modality gap between vision and language in SLT. Experimental results on two widely adopted benchmarks, RWTH-PHOENIX-Weather 2014 T and CSL-Daily, show that our method achieves results comparable to state-of-the-art and outperforms the pure-attention baseline. Additionally, this work opens a new door for future research into gloss-free SLT using text-based CTC alignment.