# Beyond a Single Reference: Training and Evaluation with Paraphrases in Sign Language Translation

arXiv: https://arxiv.org/abs/2601.21128 · HTML: https://mlsanity.com/paper/2601.21128/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [mm-Pose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNs](https://mlsanity.com/paper/1911.09592/) (2019)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus](https://mlsanity.com/paper/2407.11144/) (2024)
- [On the Evaluation Metrics for Paraphrase Generation](https://mlsanity.com/paper/2202.08479/) (2022)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation](https://mlsanity.com/paper/2408.13585/) (2024)
- [Exploring Pose-based Sign Language Translation: Ablation Studies and Attention Insights](https://mlsanity.com/paper/2507.01532/) (2025)
- [Saudi Sign Language Translation Using T5](https://mlsanity.com/paper/2510.11183/) (2025)

## Abstract (arXiv)

Most Sign Language Translation (SLT) corpora pair each signed utterance with a single written-language reference, despite the highly non-isomorphic relationship between sign and spoken languages, where multiple translations can be equally valid. This limitation constrains both model training and evaluation, particularly for n-gram-based metrics such as BLEU. In this work, we investigate the use of Large Language Models to automatically generate paraphrased variants of written-language translations as synthetic alternative references for SLT. First, we compare multiple paraphrasing strategies and models using an adapted ParaScore metric. Second, we study the impact of paraphrases on both training and evaluation of the pose-based T5 model on the YouTubeASL and How2Sign datasets. Our results show that naively incorporating paraphrases during training does not improve translation performance and can even be detrimental. In contrast, using paraphrases during evaluation leads to higher automatic scores and better alignment with human judgments. To formalize this observation, we introduce BLEUpara, an extension of BLEU that evaluates translations against multiple paraphrased references. Human evaluation confirms that BLEUpara correlates more strongly with perceived translation quality. We release all generated paraphrases, generation and evaluation code to support reproducible and more reliable evaluation of SLT systems.