# Gloss Attention for Gloss-free Sign Language Translation

arXiv: https://arxiv.org/abs/2307.07361 · HTML: https://mlsanity.com/paper/2307.07361/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 79 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)

## Built on by (most cited first, 30 of 51)

- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2405.04164/) (2024)
- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024)
- [C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval](https://mlsanity.com/paper/2408.09949/) (2024)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervision](https://mlsanity.com/paper/2405.10266/) (2024)
- [Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2411.16789/) (2024)
- [Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation](https://mlsanity.com/paper/2505.15438/) (2025)
- [Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation](https://mlsanity.com/paper/2506.00129/) (2025)
- [Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representations](https://mlsanity.com/paper/2309.12179/) (2023)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning](https://mlsanity.com/paper/2506.09473/) (2025)
- [Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2507.23575/) (2025)
- [EvSign: Sign Language Recognition and Translation with Streaming Events](https://mlsanity.com/paper/2407.12593/) (2024)
- [Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition](https://mlsanity.com/paper/2506.03615/) (2025)
- [Hierarchical Feature Alignment for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2507.06732/) (2025)
- [A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production](https://mlsanity.com/paper/2407.02854/) (2024)
- [Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2503.06484/) (2025)
- [Modelling the Distribution of Human Motion for Sign Language Assessment](https://mlsanity.com/paper/2408.10073/) (2024)
- [Understanding Co-speech Gestures in-the-wild](https://mlsanity.com/paper/2503.22668/) (2025)
- [Language Model is a Branch Predictor for Simultaneous Machine Translation](https://mlsanity.com/paper/2312.14488/) (2023)
- [A Chinese Continuous Sign Language Dataset Based on Complex Environments](https://mlsanity.com/paper/2409.11960/) (2024)
- [Exploring Pose-based Sign Language Translation: Ablation Studies and Attention Insights](https://mlsanity.com/paper/2507.01532/) (2025)
- [RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2512.07273/) (2025)
- [Diverse Sign Language Translation](https://mlsanity.com/paper/2410.19586/) (2024)
- [DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model](https://mlsanity.com/paper/2411.17248/) (2024)
- [ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation](https://mlsanity.com/paper/2504.11942/) (2025)
- [SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translation](https://mlsanity.com/paper/2507.09266/) (2025)

## Abstract (arXiv)

Most sign language translation (SLT) methods to date require the use of gloss annotations to provide additional supervision information, however, the acquisition of gloss is not easy. To solve this problem, we first perform an analysis of existing models to confirm how gloss annotations make SLT easier. We find that it can provide two aspects of information for the model, 1) it can help the model implicitly learn the location of semantic boundaries in continuous sign language videos, 2) it can help the model understand the sign language video globally. We then propose gloss attention, which enables the model to keep its attention within video segments that have the same semantics locally, just as gloss helps existing models do. Furthermore, we transfer the knowledge of sentence-to-sentence similarity from the natural language model to our gloss attention SLT network (GASLT) to help it understand sign language videos at the sentence level. Experimental results on multiple large-scale sign language datasets show that our proposed GASLT model significantly outperforms existing methods. Our code is provided in urlhttps://github.com/YinAoXiong/GASLT.