# Gloss-Free End-to-End Sign Language Translation

arXiv: https://arxiv.org/abs/2305.12876 · HTML: https://mlsanity.com/paper/2305.12876/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Open-Domain Sign Language Translation Learned from Online Video](https://mlsanity.com/paper/2205.12870/) (2022)
- [Neural Sign Language Translation based on Human Keypoint Estimation](https://mlsanity.com/paper/1811.11436/) (2018)
- [Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models](https://mlsanity.com/paper/2203.10326/) (2022)
- [Keypoint based Sign Language Translation without Glosses](https://mlsanity.com/paper/2204.10511/) (2022)

## Built on by (most cited first, 30 of 32)

- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval](https://mlsanity.com/paper/2408.09949/) (2024)
- [SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale](https://mlsanity.com/paper/2406.06907/) (2024)
- [LLaVA-SLT: Visual Language Tuning for Sign Language Translation](https://mlsanity.com/paper/2412.16524/) (2024)
- [Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representations](https://mlsanity.com/paper/2309.12179/) (2023)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Scaling Sign Language Translation](https://mlsanity.com/paper/2407.11855/) (2024)
- [CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation](https://mlsanity.com/paper/2404.11111/) (2024)
- [EvSign: Sign Language Recognition and Translation with Streaming Events](https://mlsanity.com/paper/2407.12593/) (2024)
- [Sign Language Translation with Sentence Embedding Supervision](https://mlsanity.com/paper/2510.19367/) (2025)
- [Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment](https://mlsanity.com/paper/2512.08040/) (2025)
- [A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production](https://mlsanity.com/paper/2407.02854/) (2024)
- [RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2512.07273/) (2025)
- [Diverse Sign Language Translation](https://mlsanity.com/paper/2410.19586/) (2024)
- [SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation](https://mlsanity.com/paper/2509.00030/) (2025)
- [Text2Sign Diffusion: A Generative Approach for Gloss-Free Sign Language Production](https://mlsanity.com/paper/2509.10845/) (2025)
- [Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline](https://mlsanity.com/paper/2408.10488/) (2024)
- [iLSU-T: an Open Dataset for Uruguayan Sign Language Translation](https://mlsanity.com/paper/2507.21104/) (2025)
- [Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learning](https://mlsanity.com/paper/2508.14574/) (2025)
- [FEA-SLT: A Gloss-Free End-to-End Framework for Facial-Expression-Aware Sign Language Translation](https://mlsanity.com/paper/2601.03549/) (2026)
- [SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation](https://mlsanity.com/paper/2604.18034/) (2026)
- [Towards Compact Sign Language Translation: Frame Rate and Model Size Trade-offs](https://mlsanity.com/paper/2605.09554/) (2026)
- [Towards Continuous Sign Language Conversation from Isolated Signs](https://mlsanity.com/paper/2605.14705/) (2026)
- [Direct Translation between Sign Languages](https://mlsanity.com/paper/2605.20588/) (2026)
- [VTaMo: Video-Text Alignment Model for Sign Language Translation](https://mlsanity.com/paper/2607.09126/) (2026)
- [Toward Real-Time Sentence-Level Sign Language Translation](https://mlsanity.com/paper/2607.09611/) (2026)
- [Q-BridgeNet: A Quantization Network for Cross-Lingual Sign Language Translation](https://mlsanity.com/paper/2607.11215/) (2026)
- [SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation](https://mlsanity.com/paper/2609.06296/) (2026)
- [OneSign: Unifying Sign Language Understanding Tasks with One Model](https://mlsanity.com/paper/2609.33090/) (2026)

## Abstract (arXiv)

In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations. Although intermediate representation like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities. This limits the domain coverage of translation datasets, thus handicapping real-world applications. To mitigate this problem, we design the Gloss-Free End-to-end sign language translation framework (GloFE). Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. Common concepts are extracted from the text and used as a weak form of intermediate representation. The global embedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features. In a contrastive manner, we encourage the similarity of query results between samples containing such concepts and decrease those that do not. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign. The code and model will be available at https://github.com/HenryLittle/GloFE.