# Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation

arXiv: https://arxiv.org/abs/2609.07965 · HTML: https://mlsanity.com/paper/2609.07965/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation](https://mlsanity.com/paper/2102.01818/) (2021)
- [Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation](https://mlsanity.com/paper/1706.09799/) (2017)
- [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024)
- [SimulSLT: End-to-End Simultaneous Sign Language Translation](https://mlsanity.com/paper/2112.04228/) (2021)
- [Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment](https://mlsanity.com/paper/2312.15645/) (2023)
- [Sign Language Translation with Iterative Prototype](https://mlsanity.com/paper/2308.12191/) (2023)
- [Neural Sign Language Translation by Learning Tokenization](https://mlsanity.com/paper/2002.00479/) (2020)

## Abstract (arXiv)

Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev/test. This raises concerns about whether models truly generalise or instead rely on signer-specific regularities. We conduct signer-fold cross-validation on GFSLT-VLP, GASLT, and SignCL, three leading, publicly available, gloss-free SLT models, on CSL-Daily and PHOENIX14T. Under signer-independent evaluation, performance drops sharply: on PHOENIX14T, GFSLT-VLP falls from BLEU-4 21.44 to 3.59 and ROUGE-L 42.49 to 11.89; GASLT from 15.74 to 8.26; and SignCL from 22.74 to 3.66. We also observe that in CSL-Daily many target sentences are performed by multiple signers, so common splits can place identical sentences in both training and test, inflating absolute scores by rewarding recall of recurring sentences rather than genuine generalisation. These findings indicate that signer-dependent evaluation can substantially overestimate SLT capability. We recommend: (1) adopting signer-independent protocols to ensure generalisation to unseen signers; (2) restructuring datasets to include explicit signer-independent, sentence-disjoint splits for consistent benchmarking; and (3) reporting both signer-dependent and signer-independent results together with train-test sentence overlap to improve transparency and comparability.