# Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing

arXiv: https://arxiv.org/abs/2407.01394 · HTML: https://mlsanity.com/paper/2407.01394/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 20 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 14 of 14)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [Revisiting Low-Resource Neural Machine Translation: A Case Study](https://mlsanity.com/paper/1905.11901/) (2019)
- [compare-mt: A Tool for Holistic Comparison of Language Generation Systems](https://mlsanity.com/paper/1903.07926/) (2019)
- [Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation](https://mlsanity.com/paper/2210.07054/) (2022)

## Built on by (most cited first, 3 of 3)

- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios](https://mlsanity.com/paper/2508.18183/) (2025)
- [DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation](https://mlsanity.com/paper/2609.02796/) (2026)

## Abstract (arXiv)

Sign language translation from video to spoken text presents unique challenges owing to the distinct grammar, expression nuances, and high variation of visual appearance across different speakers and contexts. The intermediate gloss annotations of videos aim to guide the translation process. In our work, we focus on em Gloss2Text translation stage and propose several advances by leveraging pre-trained large language models (LLMs), data augmentation, and novel label-smoothing loss function exploiting gloss translation ambiguities improving significantly the performance of state-of-the-art approaches. Through extensive experiments and ablation studies on the PHOENIX Weather 2014T dataset, our approach surpasses state-of-the-art performance in em Gloss2Text translation, indicating its efficacy in addressing sign language translation and suggesting promising avenues for future research and development.