# The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemes

arXiv: https://arxiv.org/abs/2310.00196 · HTML: https://mlsanity.com/paper/2310.00196/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 30 Sep 2023.

## Builds on (most cited first, 9 of 9)

- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective](https://mlsanity.com/paper/1908.08597/) (2019)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition](https://mlsanity.com/paper/2304.05934/) (2023)
- [OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages](https://mlsanity.com/paper/2110.05877/) (2021)
- [WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language](https://mlsanity.com/paper/2203.06096/) (2022)
- [Improving Sign Recognition with Phonology](https://mlsanity.com/paper/2302.05759/) (2023)

## Built on by (most cited first, 17 of 17)

- [SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction](https://mlsanity.com/paper/2411.16765/) (2024)
- [SignCLIP: Connecting Text and Sign Language by Contrastive Learning](https://mlsanity.com/paper/2407.01264/) (2024)
- [Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas](https://mlsanity.com/paper/2403.02563/) (2024)
- [Meaningful Pose-Based Sign Language Evaluation](https://mlsanity.com/paper/2510.07453/) (2025)
- [3D-LEX v1.0: 3D Lexicons for American Sign Language and Sign Language of the Netherlands](https://mlsanity.com/paper/2409.01901/) (2024)
- [The NGT200 Dataset: Geometric Multi-View Isolated Sign Recognition](https://mlsanity.com/paper/2409.15284/) (2024)
- [Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing](https://mlsanity.com/paper/2512.08094/) (2025)
- [Studying and Mitigating Biases in Sign Language Understanding Models](https://mlsanity.com/paper/2410.05206/) (2024)
- [The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge](https://mlsanity.com/paper/2411.03568/) (2024)
- [Phonological Representation Learning for Isolated Signs Improves Out-of-Vocabulary Generalization](https://mlsanity.com/paper/2509.04745/) (2025)
- [How Pragmatics Shape Articulation: A Computational Case Study in STEM ASL Discourse](https://mlsanity.com/paper/2510.23842/) (2025)
- [Representing Signs as Signs: One-Shot ISLR to Facilitate Functional Sign Language Technologies](https://mlsanity.com/paper/2502.20171/) (2025)
- [Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs](https://mlsanity.com/paper/2604.27232/) (2026)
- [Phonological Perception of Sign Language Models](https://mlsanity.com/paper/2606.28667/) (2026)
- [SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval](https://mlsanity.com/paper/2609.03695/) (2026)
- [SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation](https://mlsanity.com/paper/2609.06296/) (2026)
- [Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes](https://mlsanity.com/paper/2609.18772/) (2026)

## Abstract (arXiv)

Sign language recognition and translation technologies have the potential to increase access and inclusion of deaf signing communities, but research progress is bottlenecked by a lack of representative data. We introduce a new resource for American Sign Language (ASL) modeling, the Sem-Lex Benchmark. The Benchmark is the current largest of its kind, consisting of over 84k videos of isolated sign productions from deaf ASL signers who gave informed consent and received compensation. Human experts aligned these videos with other sign language resources including ASL-LEX, SignBank, and ASL Citizen, enabling useful expansions for sign and phonological feature recognition. We present a suite of experiments which make use of the linguistic information in ASL-LEX, evaluating the practicality and fairness of the Sem-Lex Benchmark for isolated sign recognition (ISR). We use an SL-GCN model to show that the phonological features are recognizable with 85% accuracy, and that they are effective as an auxiliary target to ISR. Learning to recognize phonological features alongside gloss results in a 6% improvement for few-shot ISR accuracy and a 2% improvement for ISR accuracy overall. Instructions for downloading the data can be found at https://github.com/leekezar/SemLex.