# signwriting-evaluation: Effective Sign Language Evaluation via SignWriting

arXiv: https://arxiv.org/abs/2410.13668 · HTML: https://mlsanity.com/paper/2410.13668/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 12 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 5 of 5)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting](https://mlsanity.com/paper/2210.05404/) (2022)
- [An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics](https://mlsanity.com/paper/2305.14998/) (2023)

## Built on by (most cited first, 1 of 1)

- [SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluation](https://mlsanity.com/paper/2509.03791/) (2025)

## Abstract (arXiv)

The lack of automatic evaluation metrics tailored for SignWriting presents a significant obstacle in developing effective transcription and translation models for signed languages. This paper introduces a comprehensive suite of evaluation metrics specifically designed for SignWriting, including adaptations of standard metrics such as textttBLEU and textttchrF, the application of textttCLIPScore to SignWriting images, and a novel symbol distance metric unique to our approach. We address the distinct challenges of evaluating single signs versus continuous signing and provide qualitative demonstrations of metric efficacy through score distribution analyses and nearest-neighbor searches within the SignBank corpus. Our findings reveal the strengths and limitations of each metric, offering valuable insights for future advancements using SignWriting. This work contributes essential tools for evaluating SignWriting models, facilitating progress in the field of sign language processing. Our code is available at urlhttps://github.com/sign-language-processing/signwriting-evaluation.