mlsanityData to 14 Oct 2026

Visual Keyword Spotting with Attention

arXiv 2110.15957 ↗First posted 29 Oct 2021cs.CV cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider the task of spotting spoken keywords in silent video sequences -- also known as visual keyword spotting. To this end, we investigate Transformer-based models that ingest two streams, a visual encoding of the video and a phonetic encoding of the keyword, and output the temporal location of the keyword if present. Our contributions are as follows: (1) We propose a novel architecture, the Transpotter, that uses full cross-modal attention between the visual and phonetic streams; (2) We show through extensive evaluations that our model outperforms the prior state-of-the-art visual keyword spotting and lip reading methods on the challenging LRW, LRS2, LRS3 datasets by a large margin; (3) We demonstrate the ability of our model to spot words under the extreme conditions of isolated mouthings in sign language videos.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 43 of its 80 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

43 of 80 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2017Localizing Moments in Video with Natural Languagecited by 663
  8. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  9. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 334
  10. 2018Deep Audio-Visual Speech Recognitioncited by 322
  11. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  12. 2016Lip Reading Sentences in the Wildcited by 228
31 more
  1. 2020Dense Regression Network for Video Groundingcited by 176
  2. 2018To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regressioncited by 157
  3. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  4. 2018Multilevel Language and Vision Integration for Text-to-Clip Retrievalcited by 128
  5. 2017Hello Edge: Keyword Spotting on Microcontrollerscited by 110
  6. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  7. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  8. 2019ExCL: Extractive Clip Localization Using Natural Language Descriptionscited by 82
  9. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 74
  10. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 68
  11. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  12. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  13. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 60
  14. 2018Large-Scale Visual Speech Recognitioncited by 57
  15. 2019Transformers with convolutional context for ASRcited by 44
  16. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  17. 2020Watch, read and lookup: learning to spot signs from multiple supervisorscited by 35
  18. 2021Read and Attend: Temporal Localisation in Sign Language Videoscited by 34
  19. 2016Trainable Frontend For Robust and Far-Field Keyword Spottingcited by 33
  20. 2017Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spottingcited by 32
  21. 2021Aligning Subtitles in Sign Language Videoscited by 26
  22. 2020Seeing wake words: Audio-visual Keyword Spottingcited by 22
  23. 2017End-to-End ASR-free Keyword Search from Speechcited by 19
  24. 2017Streaming Small-Footprint Keyword Spotting using Sequence-to-Sequence Modelscited by 19
  25. 2017Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spottingcited by 15
  26. 2018Zero-shot keyword spotting for visual speech recognition in-the-wildcited by 15
  27. 2019Small-footprint Keyword Spotting with Graph Convolutional Networkcited by 7
  28. 2015Online Keyword Spotting with a Character-Level Recurrent Neural Networkcited by 5
  29. 2016Signs in time: Encoding human motion as a temporal imagecited by 4
  30. 2016An End-to-End Architecture for Keyword Spotting and Voice Activity Detectioncited by 4
  31. 2018Sequence-to-sequence Models for Small-Footprint Keyword Spottingcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 2; 2025: 4.

11 papers.

  1. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  2. 2022Automatic dense annotation of large-vocabulary sign language videoscited by 20
  3. 2022Weakly-supervised Fingerspelling Recognition in British Sign Language Videoscited by 10
  4. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  5. 2024Towards Accurate Lip-to-Speech Synthesis in-the-Wildcited by 7
  6. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 7
  7. 2023Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Videocited by 4
  8. 2023Visually Grounded Keyword Detection and Localisation for Low-Resource Languagescited by 2
  9. 2024Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognitioncited by 0
  10. 2025Enhancing Visual Forced Alignment with Local Context-Aware Feature Extraction and Multi-Task Learningcited by 0
  11. 2025Learning Speaker-Invariant Visual Features for Lipreadingcited by 0