# Matching Latent Encoding for Audio-Text based Keyword Spotting

arXiv: https://arxiv.org/abs/2306.05245 · HTML: https://mlsanity.com/paper/2306.05245/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder](https://mlsanity.com/paper/1603.00982/) (2016)
- [Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches](https://mlsanity.com/paper/1611.02550/) (2016)
- [Query-by-Example Keyword Spotting system using Multi-head Attention and Softtriple Loss](https://mlsanity.com/paper/2102.07061/) (2021)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Multi-view Recurrent Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1611.04496/) (2016)
- [Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2206.15400/) (2022)

## Built on by (most cited first, 15 of 15)

- [Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding](https://mlsanity.com/paper/2308.06472/) (2023)
- [Open-vocabulary Keyword-spotting with Adaptive Instance Normalization](https://mlsanity.com/paper/2309.08561/) (2023)
- [Keyword-Guided Adaptation of Automatic Speech Recognition](https://mlsanity.com/paper/2406.02649/) (2024)
- [MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting](https://mlsanity.com/paper/2406.07310/) (2024)
- [Relational Proxy Loss for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2406.05314/) (2024)
- [Text-Aware Adapter for Few-Shot Keyword Spotting](https://mlsanity.com/paper/2412.18142/) (2024)
- [End-to-End User-Defined Keyword Spotting using Shifted Delta Coefficients](https://mlsanity.com/paper/2405.14489/) (2024)
- [Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2505.16735/) (2025)
- [Noise-Agnostic Multitask Whisper Training for Reducing False Alarm Errors in Call-for-Help Detection](https://mlsanity.com/paper/2501.11631/) (2025)
- [PatchDSU: Uncertainty Modeling for Out of Distribution Generalization in Keyword Spotting](https://mlsanity.com/paper/2508.03190/) (2025)
- [Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices](https://mlsanity.com/paper/2508.04857/) (2025)
- [MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2601.14012/) (2026)
- [PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting](https://mlsanity.com/paper/2603.18023/) (2026)
- [ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody](https://mlsanity.com/paper/2603.18024/) (2026)
- [Massive Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2606.11279/) (2026)

## Abstract (arXiv)

Using audio and text embeddings jointly for Keyword Spotting (KWS) has shown high-quality results, but the key challenge of how to semantically align two embeddings for multi-word keywords of different sequence lengths remains largely unsolved. In this paper, we propose an audio-text-based end-to-end model architecture for flexible keyword spotting (KWS), which builds upon learned audio and text embeddings. Our architecture uses a novel dynamic programming-based algorithm, Dynamic Sequence Partitioning (DSP), to optimally partition the audio sequence into the same length as the word-based text sequence using the monotonic alignment of spoken content. Our proposed model consists of an encoder block to get audio and text embeddings, a projector block to project individual embeddings to a common latent space, and an audio-text aligner containing a novel DSP algorithm, which aligns the audio and text embeddings to determine if the spoken content is the same as the text. Experimental results show that our DSP is more effective than other partitioning schemes, and the proposed architecture outperformed the state-of-the-art results on the public dataset in terms of Area Under the ROC Curve (AUC) and Equal-Error-Rate (EER) by 14.4 % and 28.9%, respectively.