# Watch, read and lookup: learning to spot signs from multiple supervisors

arXiv: https://arxiv.org/abs/2010.04002 · HTML: https://mlsanity.com/paper/2010.04002/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 30 of 35)

- [Open-Domain Sign Language Translation Learned from Online Video](https://mlsanity.com/paper/2205.12870/) (2022)
- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2303.12793/) (2023)
- [BBC-Oxford British Sign Language Dataset](https://mlsanity.com/paper/2111.03635/) (2021)
- [Aligning Subtitles in Sign Language Videos](https://mlsanity.com/paper/2105.02877/) (2021)
- [Sign Language Video Retrieval with Free-Form Textual Queries](https://mlsanity.com/paper/2201.02495/) (2022)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [Automatic dense annotation of large-vocabulary sign language videos](https://mlsanity.com/paper/2208.02802/) (2022)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervision](https://mlsanity.com/paper/2405.10266/) (2024)
- [Towards Online Continuous Sign Language Recognition and Translation](https://mlsanity.com/paper/2401.05336/) (2024)
- [Towards Zero-shot Sign Language Recognition](https://mlsanity.com/paper/2201.05914/) (2022)
- [Scaling up sign spotting through sign language dictionaries](https://mlsanity.com/paper/2205.04152/) (2022)
- [Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation](https://mlsanity.com/paper/2506.00129/) (2025)
- [Visual Keyword Spotting with Attention](https://mlsanity.com/paper/2110.15957/) (2021)
- [Weakly-supervised Fingerspelling Recognition in British Sign Language Videos](https://mlsanity.com/paper/2211.08954/) (2022)
- [Looking for the Signs: Identifying Isolated Sign Instances in Continuous Video Footage](https://mlsanity.com/paper/2108.04229/) (2021)
- [A Hong Kong Sign Language Corpus Collected from Sign-interpreted TV News](https://mlsanity.com/paper/2405.00980/) (2024)
- [Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for Isolated Indian Sign Language Recognition](https://mlsanity.com/paper/2407.14224/) (2024)
- [MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset](https://mlsanity.com/paper/2410.19488/) (2024)
- [Deep Understanding of Sign Language for Sign to Subtitle Alignment](https://mlsanity.com/paper/2503.03287/) (2025)
- [SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval](https://mlsanity.com/paper/2407.16394/) (2024)
- [Text-Driven 3D Hand Motion Generation from Sign Language Data](https://mlsanity.com/paper/2508.15902/) (2025)
- [Gloss Alignment Using Word Embeddings](https://mlsanity.com/paper/2308.04248/) (2023)
- [Logos as a Well-Tempered Pre-train for Sign Language Recognition](https://mlsanity.com/paper/2505.10481/) (2025)
- [Sign Spotting Disambiguation using Large Language Models](https://mlsanity.com/paper/2507.03703/) (2025)
- [Transformer with Controlled Attention for Synchronous Motion Captioning](https://mlsanity.com/paper/2409.09177/) (2024)
- [Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards](https://mlsanity.com/paper/2606.19352/) (2026)
- [Gloss-Free Representation Learning for Cross-Dataset Sign Spotting](https://mlsanity.com/paper/2608.11332/) (2026)

## Abstract (arXiv)

The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles (readily available translations of the signed content) which provide additional weak-supervision; (3) looking up words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.