# Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding

arXiv: https://arxiv.org/abs/2308.06472 · HTML: https://mlsanity.com/paper/2308.06472/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [Learning Efficient Representations for Keyword Spotting with Triplet Loss](https://mlsanity.com/paper/2101.04792/) (2021)
- [Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2206.15400/) (2022)
- [Matching Latent Encoding for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2306.05245/) (2023)
- [Metric Learning for User-defined Keyword Spotting](https://mlsanity.com/paper/2211.00439/) (2022)

## Built on by (most cited first, 10 of 10)

- [The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language](https://mlsanity.com/paper/2311.08323/) (2023)
- [Open-vocabulary Keyword-spotting with Adaptive Instance Normalization](https://mlsanity.com/paper/2309.08561/) (2023)
- [Keyword-Guided Adaptation of Automatic Speech Recognition](https://mlsanity.com/paper/2406.02649/) (2024)
- [MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting](https://mlsanity.com/paper/2406.07310/) (2024)
- [Text-Based Detection of On-Hold Scripts in Contact Center Calls](https://mlsanity.com/paper/2407.09849/) (2024)
- [Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2505.16735/) (2025)
- [CTC-aligned Audio-Text Embedding for Streaming Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2406.07923/) (2024)
- [SLiCK: Exploiting Subsequences for Length-Constrained Keyword Spotting](https://mlsanity.com/paper/2409.09067/) (2024)
- [Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices](https://mlsanity.com/paper/2508.04857/) (2025)
- [ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody](https://mlsanity.com/paper/2603.18024/) (2026)

## Abstract (arXiv)

Spotting user-defined/flexible keywords represented in text frequently uses an expensive text encoder for joint analysis with an audio encoder in an embedding space, which can suffer from heterogeneous modality representation (i.e., large mismatch) and increased complexity. In this work, we propose a novel architecture to efficiently detect arbitrary keywords based on an audio-compliant text encoder which inherently has homogeneous representation with audio embedding, and it is also much smaller than a compatible text encoder. Our text encoder converts the text to phonemes using a grapheme-to-phoneme (G2P) model, and then to an embedding using representative phoneme vectors, extracted from the paired audio encoder on rich speech datasets. We further augment our method with confusable keyword generation to develop an audio-text embedding verifier with strong discriminative power. Experimental results show that our scheme outperforms the state-of-the-art results on Libriphrase hard dataset, increasing Area Under the ROC Curve (AUC) metric from 84.21% to 92.7% and reducing Equal-Error-Rate (EER) metric from 23.36% to 14.4%.