# Improving Gloss-free Sign Language Translation by Reducing Representation Density

arXiv: https://arxiv.org/abs/2405.14312 · HTML: https://mlsanity.com/paper/2405.14312/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2405.04164/) (2024)
- [Gloss Attention for Gloss-free Sign Language Translation](https://mlsanity.com/paper/2307.07361/) (2023)
- [Global-local Enhancement Network for NMFs-aware Sign Language Recognition](https://mlsanity.com/paper/2008.10428/) (2020)
- [Context-Enhanced Stereo Transformer](https://mlsanity.com/paper/2210.11719/) (2022)
- [SLTUNET: A Simple Unified Model for Sign Language Translation](https://mlsanity.com/paper/2305.01778/) (2023)
- [CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2303.12793/) (2023)
- [Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network](https://mlsanity.com/paper/2111.07258/) (2021)
- [Sign Language Translation with Iterative Prototype](https://mlsanity.com/paper/2308.12191/) (2023)
- [Cross-modality Data Augmentation for End-to-End Sign Language Translation](https://mlsanity.com/paper/2305.11096/) (2023)
- [XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2403.01560/) (2024)
- [A Token-level Contrastive Framework for Sign Language Translation](https://mlsanity.com/paper/2204.04916/) (2022)
- [Interactive Interior Design Recommendation via Coarse-to-fine Multimodal Reinforcement Learning](https://mlsanity.com/paper/2310.07287/) (2023)
- [Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation](https://mlsanity.com/paper/2210.07054/) (2022)
- [Cloth2Body: Generating 3D Human Body Mesh from 2D Clothing](https://mlsanity.com/paper/2309.16189/) (2023)

## Built on by (most cited first, 30 of 31)

- [T*: Re-thinking Temporal Search for Long-Form Video Understanding](https://mlsanity.com/paper/2504.02259/) (2025)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [SignCLIP: Connecting Text and Sign Language by Contrastive Learning](https://mlsanity.com/paper/2407.01264/) (2024)
- [LLaVA-SLT: Visual Language Tuning for Sign Language Translation](https://mlsanity.com/paper/2412.16524/) (2024)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2507.23575/) (2025)
- [Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition](https://mlsanity.com/paper/2506.03615/) (2025)
- [Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment](https://mlsanity.com/paper/2512.08040/) (2025)
- [Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2503.06484/) (2025)
- [SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluation](https://mlsanity.com/paper/2509.03791/) (2025)
- [MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation](https://mlsanity.com/paper/2601.19577/) (2026)
- [Exploring Pose-based Sign Language Translation: Ablation Studies and Attention Insights](https://mlsanity.com/paper/2507.01532/) (2025)
- [SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translation](https://mlsanity.com/paper/2507.09266/) (2025)
- [Contrastive Pretraining with Dual Visual Encoders for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2507.10306/) (2025)
- [Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio](https://mlsanity.com/paper/2508.20476/) (2025)
- [SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation](https://mlsanity.com/paper/2606.28626/) (2026)
- [SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs](https://mlsanity.com/paper/2608.09006/) (2026)
- [Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline](https://mlsanity.com/paper/2408.10488/) (2024)
- [Beyond Words: AuralLLM and SignMST-C for Sign Language Production and Bidirectional Accessibility](https://mlsanity.com/paper/2501.00765/) (2025)
- [Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learning](https://mlsanity.com/paper/2508.14574/) (2025)
- [SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation](https://mlsanity.com/paper/2604.18034/) (2026)
- [Selective Contrastive Learning For Gloss Free Sign Language Translation](https://mlsanity.com/paper/2604.22374/) (2026)
- [Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching](https://mlsanity.com/paper/2606.11925/) (2026)
- [DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2607.27614/) (2026)
- [Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production](https://mlsanity.com/paper/2608.09045/) (2026)
- [ODE-Based Transformer Decoders for Iterative Sign Language Translation](https://mlsanity.com/paper/2608.11352/) (2026)
- [Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks](https://mlsanity.com/paper/2609.03734/) (2026)
- [Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation](https://mlsanity.com/paper/2609.07965/) (2026)
- [Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos](https://mlsanity.com/paper/2609.31148/) (2026)
- [OneSign: Unifying Sign Language Understanding Tasks with One Model](https://mlsanity.com/paper/2609.33090/) (2026)

## Abstract (arXiv)

Gloss-free sign language translation (SLT) aims to develop well-performing SLT systems with no requirement for the costly gloss annotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify a representation density problem that could be a bottleneck in restricting the performance of gloss-free SLT. Specifically, the representation density problem describes that the visual representations of semantically distinct sign gestures tend to be closely packed together in feature space, which makes gloss-free methods struggle with distinguishing different sign gestures and suffer from a sharp performance drop. To address the representation density problem, we introduce a simple but effective contrastive learning strategy, namely SignCL, which encourages gloss-free models to learn more discriminative feature representation in a self-supervised manner. Our experiments demonstrate that the proposed SignCL can significantly reduce the representation density and improve performance across various translation frameworks. Specifically, SignCL achieves a significant improvement in BLEU score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset by 39% and 46%, respectively, without any increase of model parameters. Compared to Sign2GPT, a state-of-the-art method based on large-scale pre-trained vision and language models, SignCL achieves better performance with only 35% of its parameters. Implementation and Checkpoints are available at https://github.com/JinhuiYE/SignCL.