# Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting

arXiv: https://arxiv.org/abs/2206.15400 · HTML: https://mlsanity.com/paper/2206.15400/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 30 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [Temporal Convolution for Real-time Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1904.03814/) (2019)
- [A scalable noisy speech dataset and online subjective test framework](https://mlsanity.com/paper/1909.08050/) (2019)
- [Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation](https://mlsanity.com/paper/2104.02775/) (2021)
- [Query-by-Example Keyword Spotting system using Multi-head Attention and Softtriple Loss](https://mlsanity.com/paper/2102.07061/) (2021)
- [Seeing wake words: Audio-visual Keyword Spotting](https://mlsanity.com/paper/2009.01225/) (2020)
- [Query-by-example on-device keyword spotting](https://mlsanity.com/paper/1910.05171/) (2019)
- [DONUT: CTC-based Query-by-Example Keyword Spotting](https://mlsanity.com/paper/1811.10736/) (2018)
- [Small-footprint Keyword Spotting with Graph Convolutional Network](https://mlsanity.com/paper/1912.05124/) (2019)
- [End-to-End Open Vocabulary Keyword Search](https://mlsanity.com/paper/2108.10357/) (2021)

## Built on by (most cited first, 20 of 20)

- [Matching Latent Encoding for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2306.05245/) (2023)
- [Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding](https://mlsanity.com/paper/2308.06472/) (2023)
- [Open-vocabulary Keyword-spotting with Adaptive Instance Normalization](https://mlsanity.com/paper/2309.08561/) (2023)
- [A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2309.09552/) (2023)
- [Keyword-Guided Adaptation of Automatic Speech Recognition](https://mlsanity.com/paper/2406.02649/) (2024)
- [MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting](https://mlsanity.com/paper/2406.07310/) (2024)
- [Improving Acoustic Word Embeddings through Correspondence Training of Self-supervised Speech Representations](https://mlsanity.com/paper/2403.08738/) (2024)
- [Relational Proxy Loss for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2406.05314/) (2024)
- [End-to-End User-Defined Keyword Spotting using Shifted Delta Coefficients](https://mlsanity.com/paper/2405.14489/) (2024)
- [Text-Based Detection of On-Hold Scripts in Contact Center Calls](https://mlsanity.com/paper/2407.09849/) (2024)
- [Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2505.16735/) (2025)
- [Improving Small Footprint Few-shot Keyword Spotting with Supervision on Auxiliary Data](https://mlsanity.com/paper/2309.00647/) (2023)
- [SLiCK: Exploiting Subsequences for Length-Constrained Keyword Spotting](https://mlsanity.com/paper/2409.09067/) (2024)
- [Noise-Agnostic Multitask Whisper Training for Reducing False Alarm Errors in Call-for-Help Detection](https://mlsanity.com/paper/2501.11631/) (2025)
- [PatchDSU: Uncertainty Modeling for Out of Distribution Generalization in Keyword Spotting](https://mlsanity.com/paper/2508.03190/) (2025)
- [Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices](https://mlsanity.com/paper/2508.04857/) (2025)
- [MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2601.14012/) (2026)
- [PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting](https://mlsanity.com/paper/2603.18023/) (2026)
- [ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody](https://mlsanity.com/paper/2603.18024/) (2026)
- [Massive Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2606.11279/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel end-to-end user-defined keyword spotting method that utilizes linguistically corresponding patterns between speech and text sequences. Unlike previous approaches requiring speech keyword enrollment, our method compares input queries with an enrolled text keyword sequence. To place the audio and text representations within a common latent space, we adopt an attention-based cross-modal matching approach that is trained in an end-to-end manner with monotonic matching loss and keyword classification loss. We also utilize a de-noising loss for the acoustic embedding network to improve robustness in noisy environments. Additionally, we introduce the LibriPhrase dataset, a new short-phrase dataset based on LibriSpeech for efficiently training keyword spotting models. Our proposed method achieves competitive results on various evaluation sets compared to other single-modal and cross-modal baselines.