# Fingerspelling Detection in American Sign Language

arXiv: https://arxiv.org/abs/2104.01291 · HTML: https://mlsanity.com/paper/2104.01291/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 59 references that are in the map.
- First posted on arXiv 3 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition](https://mlsanity.com/paper/2002.03187/) (2020)
- [Transferring Cross-domain Knowledge for Video Sign Language Recognition](https://mlsanity.com/paper/2003.03703/) (2020)
- [Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1707.03985/) (2017)
- [Real-Time Sign Language Detection using Human Pose Estimation](https://mlsanity.com/paper/2008.04637/) (2020)
- [Fingerspelling recognition in the wild with iterative visual attention](https://mlsanity.com/paper/1908.10546/) (2019)
- [American Sign Language fingerspelling recognition in the wild](https://mlsanity.com/paper/1810.11438/) (2018)
- [MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/2006.09220/) (2020)
- [Sign language segmentation with temporal convolutional networks](https://mlsanity.com/paper/2011.12986/) (2020)
- [Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation](https://mlsanity.com/paper/1609.07876/) (2016)

## Built on by (most cited first, 7 of 7)

- [Open-Domain Sign Language Translation Learned from Online Video](https://mlsanity.com/paper/2205.12870/) (2022)
- [Weakly-supervised Fingerspelling Recognition in British Sign Language Videos](https://mlsanity.com/paper/2211.08954/) (2022)
- [Searching for fingerspelled content in American Sign Language](https://mlsanity.com/paper/2203.13291/) (2022)
- [Fingerspelling PoseNet: Enhancing Fingerspelling Translation with Pose-Based Transformer Models](https://mlsanity.com/paper/2311.12128/) (2023)
- [CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language](https://mlsanity.com/paper/2604.22367/) (2026)
- [iLSU-T: an Open Dataset for Uruguayan Sign Language Translation](https://mlsanity.com/paper/2507.21104/) (2025)
- [OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis](https://mlsanity.com/paper/2602.22949/) (2026)

## Abstract (arXiv)

Fingerspelling, in which words are signed letter by letter, is an important component of American Sign Language. Most previous work on automatic fingerspelling recognition has assumed that the boundaries of fingerspelling regions in signing videos are known beforehand. In this paper, we consider the task of fingerspelling detection in raw, untrimmed sign language videos. This is an important step towards building real-world fingerspelling recognition systems. We propose a benchmark and a suite of evaluation metrics, some of which reflect the effect of detection on the downstream fingerspelling recognition task. In addition, we propose a new model that learns to detect fingerspelling via multi-task training, incorporating pose estimation and fingerspelling recognition (transcription) along with detection, and compare this model to several alternatives. The model outperforms all alternative approaches across all metrics, establishing a state of the art on the benchmark.