# Seeing wake words: Audio-visual Keyword Spotting

arXiv: https://arxiv.org/abs/2009.01225 · HTML: https://mlsanity.com/paper/2009.01225/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2020.

## Built on by (most cited first, 22 of 22)

- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Read and Attend: Temporal Localisation in Sign Language Videos](https://mlsanity.com/paper/2103.16481/) (2021)
- [BBC-Oxford British Sign Language Dataset](https://mlsanity.com/paper/2111.03635/) (2021)
- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [Sign Language Video Retrieval with Free-Form Textual Queries](https://mlsanity.com/paper/2201.02495/) (2022)
- [Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2206.15400/) (2022)
- [Automatic dense annotation of large-vocabulary sign language videos](https://mlsanity.com/paper/2208.02802/) (2022)
- [MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition](https://mlsanity.com/paper/2303.05309/) (2023)
- [Scaling up sign spotting through sign language dictionaries](https://mlsanity.com/paper/2205.04152/) (2022)
- [Visual Keyword Spotting with Attention](https://mlsanity.com/paper/2110.15957/) (2021)
- [Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages](https://mlsanity.com/paper/2306.07744/) (2023)
- [Weakly-supervised Fingerspelling Recognition in British Sign Language Videos](https://mlsanity.com/paper/2211.08954/) (2022)
- [VGGSounder: Audio-Visual Evaluations for Foundation Models](https://mlsanity.com/paper/2508.08237/) (2025)
- [AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation](https://mlsanity.com/paper/2504.20629/) (2025)
- [Searching for fingerspelled content in American Sign Language](https://mlsanity.com/paper/2203.13291/) (2022)
- [A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2309.09552/) (2023)
- [Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video](https://mlsanity.com/paper/2303.08670/) (2023)
- [Toward American Sign Language Processing in the Real World: Data, Tasks, and Methods](https://mlsanity.com/paper/2308.12419/) (2023)
- [A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning](https://mlsanity.com/paper/2202.08509/) (2022)
- [Enhancing Visual Forced Alignment with Local Context-Aware Feature Extraction and Multi-Task Learning](https://mlsanity.com/paper/2503.03286/) (2025)
- [SignMatch: Matching Dictionary Signs to Continuous Sign Language Video](https://mlsanity.com/paper/2609.01886/) (2026)

## Abstract (arXiv)

The goal of this work is to automatically determine whether and when a word of interest is spoken by a talking face, with or without the audio. We propose a zero-shot method suitable for in the wild videos. Our key contributions are: (1) a novel convolutional architecture, KWS-Net, that uses a similarity map intermediate representation to separate the task into (i) sequence matching, and (ii) pattern detection, to decide whether the word is there and when; (2) we demonstrate that if audio is available, visual keyword spotting improves the performance both for a clean and noisy audio signal. Finally, (3) we show that our method generalises to other languages, specifically French and German, and achieves a comparable performance to English with less language specific data, by fine-tuning the network pre-trained on English. The method exceeds the performance of the previous state-of-the-art visual keyword spotting architecture when trained and tested on the same benchmark, and also that of a state-of-the-art lip reading method.