# An Open-Source American Sign Language Fingerspell Recognition and Semantic Pose Retrieval Interface

arXiv: https://arxiv.org/abs/2408.09311 · HTML: https://mlsanity.com/paper/2408.09311/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 17 Aug 2024.

## Builds on (most cited first, 10 of 10)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [NeuSpell: A Neural Spelling Correction Toolkit](https://mlsanity.com/paper/2010.11085/) (2020)
- [Real-Time Sign Language Detection using Human Pose Estimation](https://mlsanity.com/paper/2008.04637/) (2020)
- [Fingerspelling recognition in the wild with iterative visual attention](https://mlsanity.com/paper/1908.10546/) (2019)
- [American Sign Language fingerspelling recognition in the wild](https://mlsanity.com/paper/1810.11438/) (2018)
- [Sign language segmentation with temporal convolutional networks](https://mlsanity.com/paper/2011.12986/) (2020)
- [Linguistically Motivated Sign Language Segmentation](https://mlsanity.com/paper/2310.13960/) (2023)
- [An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation](https://mlsanity.com/paper/2305.17714/) (2023)

## Abstract (arXiv)

This paper introduces an open-source interface for American Sign Language fingerspell recognition and semantic pose retrieval, aimed to serve as a stepping stone towards more advanced sign language translation systems. Utilizing a combination of convolutional neural networks and pose estimation models, the interface provides two modular components: a recognition module for translating ASL fingerspelling into spoken English and a production module for converting spoken English into ASL pose sequences. The system is designed to be highly accessible, user-friendly, and capable of functioning in real-time under varying environmental conditions like backgrounds, lighting, skin tones, and hand sizes. We discuss the technical details of the model architecture, application in the wild, as well as potential future enhancements for real-world consumer applications.