# Weakly-supervised Fingerspelling Recognition in British Sign Language Videos

arXiv: https://arxiv.org/abs/2211.08954 · HTML: https://mlsanity.com/paper/2211.08954/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [Transferring Cross-domain Knowledge for Video Sign Language Recognition](https://mlsanity.com/paper/2003.03703/) (2020)
- [Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1707.03985/) (2017)
- [Watch, read and lookup: learning to spot signs from multiple supervisors](https://mlsanity.com/paper/2010.04002/) (2020)
- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [Fingerspelling recognition in the wild with iterative visual attention](https://mlsanity.com/paper/1908.10546/) (2019)
- [Aligning Subtitles in Sign Language Videos](https://mlsanity.com/paper/2105.02877/) (2021)
- [Seeing wake words: Audio-visual Keyword Spotting](https://mlsanity.com/paper/2009.01225/) (2020)
- [American Sign Language fingerspelling recognition in the wild](https://mlsanity.com/paper/1810.11438/) (2018)
- [Automatic dense annotation of large-vocabulary sign language videos](https://mlsanity.com/paper/2208.02802/) (2022)
- [Zero-shot keyword spotting for visual speech recognition in-the-wild](https://mlsanity.com/paper/1807.08469/) (2018)
- [Visual Keyword Spotting with Attention](https://mlsanity.com/paper/2110.15957/) (2021)
- [Fingerspelling Detection in American Sign Language](https://mlsanity.com/paper/2104.01291/) (2021)
- [Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation](https://mlsanity.com/paper/1609.07876/) (2016)
- [Searching for fingerspelled content in American Sign Language](https://mlsanity.com/paper/2203.13291/) (2022)
- [Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition](https://mlsanity.com/paper/1710.03255/) (2017)

## Built on by (most cited first, 10 of 10)

- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [SignCLIP: Connecting Text and Sign Language by Contrastive Learning](https://mlsanity.com/paper/2407.01264/) (2024)
- [A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervision](https://mlsanity.com/paper/2405.10266/) (2024)
- [Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment](https://mlsanity.com/paper/2512.08040/) (2025)
- [Fingerspelling PoseNet: Enhancing Fingerspelling Translation with Pose-Based Transformer Models](https://mlsanity.com/paper/2311.12128/) (2023)
- [Understanding Co-speech Gestures in-the-wild](https://mlsanity.com/paper/2503.22668/) (2025)
- [Training Strategies for Isolated Sign Language Recognition](https://mlsanity.com/paper/2412.11553/) (2024)
- [Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio](https://mlsanity.com/paper/2508.20476/) (2025)
- [Recognising BSL Fingerspelling in Continuous Signing Sequences](https://mlsanity.com/paper/2603.19523/) (2026)
- [Machine Translation for Sign Languages](https://mlsanity.com/paper/2610.00881/) (2026)

## Abstract (arXiv)

The goal of this work is to detect and recognize sequences of letters signed using fingerspelling in British Sign Language (BSL). Previous fingerspelling recognition methods have not focused on BSL, which has a very different signing alphabet (e.g., two-handed instead of one-handed) to American Sign Language (ASL). They also use manual annotations for training. In contrast to previous methods, our method only uses weak annotations from subtitles for training. We localize potential instances of fingerspelling using a simple feature similarity method, then automatically annotate these instances by querying subtitle words and searching for corresponding mouthing cues from the signer. We propose a Transformer architecture adapted to this task, with a multiple-hypothesis CTC loss function to learn from alternative annotation possibilities. We employ a multi-stage training approach, where we make use of an initial version of our trained model to extend and enhance our training data before re-training again to achieve better performance. Through extensive evaluations, we verify our method for automatic annotation and our model architecture. Moreover, we provide a human expert annotated test set of 5K video clips for evaluating BSL fingerspelling recognition methods to support sign language research.