# Personalized Keyword Spotting through Multi-task Learning

arXiv: https://arxiv.org/abs/2206.13708 · HTML: https://mlsanity.com/paper/2206.13708/

## Key facts

- Cited by 7 papers in the map (data through 7 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 28 Jun 2022.

## Builds on (most cited first, 19 of 19)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [Prototype Rectification for Few-Shot Learning](https://mlsanity.com/paper/1911.10713/) (2019)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [Temporal Convolution for Real-time Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1904.03814/) (2019)
- [Streaming keyword spotting on mobile devices](https://mlsanity.com/paper/2005.06720/) (2020)
- [Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference](https://mlsanity.com/paper/2007.12540/) (2020)
- [Broadcasted Residual Learning for Efficient Keyword Spotting](https://mlsanity.com/paper/2106.04140/) (2021)
- [Query-by-Example Keyword Spotting system using Multi-head Attention and Softtriple Loss](https://mlsanity.com/paper/2102.07061/) (2021)
- [Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutions](https://mlsanity.com/paper/1911.02086/) (2019)
- [Query-by-example on-device keyword spotting](https://mlsanity.com/paper/1910.05171/) (2019)
- [A Cascade Architecture for Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1712.03603/) (2017)
- [Depthwise Separable Convolutional ResNet with Squeeze-and-Excitation Blocks for Small-footprint Keyword Spotting](https://mlsanity.com/paper/2004.12200/) (2020)
- [AutoKWS: Keyword Spotting with Differentiable Architecture Search](https://mlsanity.com/paper/2009.03658/) (2020)
- [Multi-Task Network for Noise-Robust Keyword Spotting and Speaker Verification using CTC-based Soft VAD and Global Query Attention](https://mlsanity.com/paper/2005.03867/) (2020)
- [Learning Acoustic Word Embeddings with Temporal Context for Query-by-Example Speech Search](https://mlsanity.com/paper/1806.03621/) (2018)
- [An End-to-End Text-independent Speaker Verification Framework with a Keyword Adversarial Network](https://mlsanity.com/paper/1908.02612/) (2019)

## Built on by (most cited first, 7 of 7)

- [Few-Shot Open-Set Learning for On-Device Customization of KeyWord Spotting Systems](https://mlsanity.com/paper/2306.02161/) (2023)
- [Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensors](https://mlsanity.com/paper/2408.12481/) (2024)
- [Latent Phrase Matching for Dysarthric Speech](https://mlsanity.com/paper/2306.05446/) (2023)
- [Personalizing Keyword Spotting with Speaker Information](https://mlsanity.com/paper/2311.03419/) (2023)
- [Improving Small Footprint Few-shot Keyword Spotting with Supervision on Auxiliary Data](https://mlsanity.com/paper/2309.00647/) (2023)
- [OnDA: On-device Channel Pruning for Efficient Personalized Keyword Spotting](https://mlsanity.com/paper/2603.02247/) (2026)
- [PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting](https://mlsanity.com/paper/2603.18023/) (2026)

## Abstract (arXiv)

Keyword spotting (KWS) plays an essential role in enabling speech-based user interaction on smart devices, and conventional KWS (C-KWS) approaches have concentrated on detecting user-agnostic pre-defined keywords. However, in practice, most user interactions come from target users enrolled in the device which motivates to construct personalized keyword spotting. We design two personalized KWS tasks; (1) Target user Biased KWS (TB-KWS) and (2) Target user Only KWS (TO-KWS). To solve the tasks, we propose personalized keyword spotting through multi-task learning (PK-MTL) that consists of multi-task learning and task-adaptation. First, we introduce applying multi-task learning on keyword spotting and speaker verification to leverage user information to the keyword spotting system. Next, we design task-specific scoring functions to adapt to the personalized KWS tasks thoroughly. We evaluate our framework on conventional and personalized scenarios, and the results show that PK-MTL can dramatically reduce the false alarm rate, especially in various practical scenarios.