# Attention-Based Keyword Localisation in Speech using Visual Grounding

arXiv: https://arxiv.org/abs/2106.08859 · HTML: https://mlsanity.com/paper/2106.08859/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Sequence-to-Sequence Models Can Directly Translate Foreign Speech](https://mlsanity.com/paper/1703.08581/) (2017)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Learning Word-Like Units from Joint Audio-Visual Analysis](https://mlsanity.com/paper/1701.07481/) (2017)
- [Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech](https://mlsanity.com/paper/1804.03052/) (2018)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Semantic speech retrieval with a visually grounded model of untranscribed speech](https://mlsanity.com/paper/1710.01949/) (2017)
- [Visually grounded cross-lingual keyword spotting in speech](https://mlsanity.com/paper/1806.05030/) (2018)
- [Visually grounded learning of keyword prediction from untranscribed speech](https://mlsanity.com/paper/1703.08136/) (2017)
- [Towards Visually Grounded Sub-Word Speech Unit Discovery](https://mlsanity.com/paper/1902.08213/) (2019)
- [Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanese](https://mlsanity.com/paper/1902.03052/) (2019)
- [Multimodal One-Shot Learning of Speech and Images](https://mlsanity.com/paper/1811.03875/) (2018)
- [Semantic query-by-example speech search using visual grounding](https://mlsanity.com/paper/1904.07078/) (2019)
- [From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning](https://mlsanity.com/paper/1610.03342/) (2016)
- [On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrieval](https://mlsanity.com/paper/1904.10947/) (2019)
- [Unsupervised vs. transfer learning for multimodal one-shot matching of speech and images](https://mlsanity.com/paper/2008.06258/) (2020)
- [Towards localisation of keywords in speech using weak supervision](https://mlsanity.com/paper/2012.07396/) (2020)

## Built on by (most cited first, 13 of 13)

- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022)
- [Fast-Slow Transformer for Visually Grounding Speech](https://mlsanity.com/paper/2109.08186/) (2021)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual grounding](https://mlsanity.com/paper/2210.04600/) (2022)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)
- [Towards visually prompted keyword localisation for zero-resource spoken languages](https://mlsanity.com/paper/2210.06229/) (2022)
- [Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples](https://mlsanity.com/paper/2303.17517/) (2023)
- [Visually grounded few-shot word learning in low-resource settings](https://mlsanity.com/paper/2306.11371/) (2023)
- [Visually grounded few-shot word acquisition with fewer shots](https://mlsanity.com/paper/2305.15937/) (2023)
- [Connecting Speech to Words through Images](https://mlsanity.com/paper/2606.16807/) (2026)
- [Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings](https://mlsanity.com/paper/2409.06013/) (2024)
- [Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding](https://mlsanity.com/paper/2608.26925/) (2026)

## Abstract (arXiv)

Visually grounded speech models learn from images paired with spoken captions. By tagging images with soft text labels using a trained visual classifier with a fixed vocabulary, previous work has shown that it is possible to train a model that can detect whether a particular text keyword occurs in speech utterances or not. Here we investigate whether visually grounded speech models can also do keyword localisation: predicting where, within an utterance, a given textual keyword occurs without any explicit text-based or alignment supervision. We specifically consider whether incorporating attention into a convolutional model is beneficial for localisation. Although absolute localisation performance with visually supervised models is still modest (compared to using unordered bag-of-word text labels for supervision), we show that attention provides a large gain in performance over previous visually grounded models. As in many other speech-image studies, we find that many of the incorrect localisations are due to semantic confusions, e.g. locating the word 'backstroke' for the query keyword 'swimming'.