# Zero-shot keyword spotting for visual speech recognition in-the-wild

arXiv: https://arxiv.org/abs/1807.08469 · HTML: https://mlsanity.com/paper/1807.08469/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2018.

## Built on by (most cited first, 15 of 15)

- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [BBC-Oxford British Sign Language Dataset](https://mlsanity.com/paper/2111.03635/) (2021)
- [Sign Language Video Retrieval with Free-Form Textual Queries](https://mlsanity.com/paper/2201.02495/) (2022)
- [Automatic dense annotation of large-vocabulary sign language videos](https://mlsanity.com/paper/2208.02802/) (2022)
- [Scaling up sign spotting through sign language dictionaries](https://mlsanity.com/paper/2205.04152/) (2022)
- [Visual Keyword Spotting with Attention](https://mlsanity.com/paper/2110.15957/) (2021)
- [VALLR: Visual ASR Language Model for Lip Reading](https://mlsanity.com/paper/2503.21408/) (2025)
- [Weakly-supervised Fingerspelling Recognition in British Sign Language Videos](https://mlsanity.com/paper/2211.08954/) (2022)
- [Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video](https://mlsanity.com/paper/2303.08670/) (2023)
- [Exploring Deep Learning for Joint Audio-Visual Lip Biometrics](https://mlsanity.com/paper/2104.08510/) (2021)
- [Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition](https://mlsanity.com/paper/2402.13004/) (2024)
- [Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions](https://mlsanity.com/paper/2502.00464/) (2025)
- [PatchDSU: Uncertainty Modeling for Out of Distribution Generalization in Keyword Spotting](https://mlsanity.com/paper/2508.03190/) (2025)
- [SignMatch: Matching Dictionary Signs to Continuous Sign Language Video](https://mlsanity.com/paper/2609.01886/) (2026)
- [Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos](https://mlsanity.com/paper/2609.31148/) (2026)

## Abstract (arXiv)

Visual keyword spotting (KWS) is the problem of estimating whether a text query occurs in a given recording using only video information. This paper focuses on visual KWS for words unseen during training, a real-world, practical setting which so far has received no attention by the community. To this end, we devise an end-to-end architecture comprising (a) a state-of-the-art visual feature extractor based on spatiotemporal Residual Networks, (b) a grapheme-to-phoneme model based on sequence-to-sequence neural networks, and (c) a stack of recurrent neural networks which learn how to correlate visual features with the keyword representation. Different to prior works on KWS, which try to learn word representations merely from sequences of graphemes (i.e. letters), we propose the use of a grapheme-to-phoneme encoder-decoder model which learns how to map words to their pronunciation. We demonstrate that our system obtains very promising visual-only KWS results on the challenging LRS2 database, for keywords unseen during training. We also show that our system outperforms a baseline which addresses KWS via automatic speech recognition (ASR), while it drastically improves over other recently proposed ASR-free KWS methods.