# PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting

arXiv: https://arxiv.org/abs/2603.18023 · HTML: https://mlsanity.com/paper/2603.18023/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [Temporal Convolution for Real-time Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1904.03814/) (2019)
- [Broadcasted Residual Learning for Efficient Keyword Spotting](https://mlsanity.com/paper/2106.04140/) (2021)
- [Streaming keyword spotting on mobile devices](https://mlsanity.com/paper/2005.06720/) (2020)
- [A neural attention model for speech command recognition](https://mlsanity.com/paper/1808.08929/) (2018)
- [SphereFace2: Binary Classification is All You Need for Deep Face Recognition](https://mlsanity.com/paper/2108.01513/) (2021)
- [MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition](https://mlsanity.com/paper/2004.08531/) (2020)
- [Deep convolutional acoustic word embeddings using word-pair side information](https://mlsanity.com/paper/1510.01032/) (2015)
- [Query-by-Example Keyword Spotting system using Multi-head Attention and Softtriple Loss](https://mlsanity.com/paper/2102.07061/) (2021)
- [Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings](https://mlsanity.com/paper/1706.03818/) (2017)
- [Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2206.15400/) (2022)
- [Noisy Differentiable Architecture Search](https://mlsanity.com/paper/2005.03566/) (2020)
- [DONUT: CTC-based Query-by-Example Keyword Spotting](https://mlsanity.com/paper/1811.10736/) (2018)
- [Small-Footprint Keyword Spotting with Multi-Scale Temporal Convolution](https://mlsanity.com/paper/2010.09960/) (2020)
- [Matching Latent Encoding for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2306.05245/) (2023)
- [PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords](https://mlsanity.com/paper/2308.16511/) (2023)
- [AutoKWS: Keyword Spotting with Differentiable Architecture Search](https://mlsanity.com/paper/2009.03658/) (2020)
- [Multi-Task Network for Noise-Robust Keyword Spotting and Speaker Verification using CTC-based Soft VAD and Global Query Attention](https://mlsanity.com/paper/2005.03867/) (2020)
- [Personalized Keyword Spotting through Multi-task Learning](https://mlsanity.com/paper/2206.13708/) (2022)
- [Metric Learning for User-defined Keyword Spotting](https://mlsanity.com/paper/2211.00439/) (2022)
- [Additional Shared Decoder on Siamese Multi-view Encoders for Learning Acoustic Word Embeddings](https://mlsanity.com/paper/1910.00341/) (2019)
- [Multilingual Query-by-Example Keyword Spotting with Metric Learning and Phoneme-to-Embedding Mapping](https://mlsanity.com/paper/2304.09585/) (2023)
- [Contrastive Learning With Audio Discrimination For Customizable Keyword Spotting In Continuous Speech](https://mlsanity.com/paper/2401.06485/) (2024)

## Abstract (arXiv)

As advancements in technologies like Internet of Things (IoT), Automatic Speech Recognition (ASR), Speaker Verification (SV), and Text-to-Speech (TTS) lead to increased usage of intelligent voice assistants, the demand for privacy and personalization has escalated. In this paper, we introduce a multi-task learning framework for personalized, customizable open-vocabulary Keyword Spotting (PCOV-KWS). This framework employs a lightweight network to simultaneously perform Keyword Spotting (KWS) and SV to address personalized KWS requirements. We have integrated a training criterion distinct from softmax-based loss, transforming multi-class classification into multiple binary classifications, which eliminates inter-category competition, while an optimization strategy for multi-task loss weighting is employed during training. We evaluated our PCOV-KWS system in multiple datasets, demonstrating that it outperforms the baselines in evaluation results, while also requiring fewer parameters and lower computational resources.