# Personalizing Keyword Spotting with Speaker Information

arXiv: https://arxiv.org/abs/2311.03419 · HTML: https://mlsanity.com/paper/2311.03419/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [End-to-End Streaming Keyword Spotting](https://mlsanity.com/paper/1812.02802/) (2018)
- [Personal VAD: Speaker-Conditioned Voice Activity Detection](https://mlsanity.com/paper/1908.04284/) (2019)
- [Speaker adaptation for Wav2vec2 based dysarthric ASR](https://mlsanity.com/paper/2204.00770/) (2022)
- [Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1705.02411/) (2017)
- [Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition](https://mlsanity.com/paper/2204.03793/) (2022)
- [Learning To Detect Keyword Parts And Whole By Smoothed Max Pooling](https://mlsanity.com/paper/2001.09246/) (2020)
- [Dr-Vectors: Decision Residual Networks and an Improved Loss for Speaker Recognition](https://mlsanity.com/paper/2104.01989/) (2021)
- [Noisy student-teacher training for robust keyword spotting](https://mlsanity.com/paper/2106.01604/) (2021)
- [Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering](https://mlsanity.com/paper/2210.13690/) (2022)
- [Personalized Keyword Spotting through Multi-task Learning](https://mlsanity.com/paper/2206.13708/) (2022)
- [Version Control of Speaker Recognition Systems](https://mlsanity.com/paper/2007.12069/) (2020)
- [Progressive Voice Trigger Detection: Accuracy vs Latency](https://mlsanity.com/paper/2010.15446/) (2020)
- [Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech](https://mlsanity.com/paper/2202.12163/) (2022)
- [Compression of Acoustic Event Detection Models With Quantized Distillation](https://mlsanity.com/paper/1907.00873/) (2019)
- [Locale Encoding For Scalable Multilingual Keyword Spotting Models](https://mlsanity.com/paper/2302.12961/) (2023)
- [Exploring Sequence-to-Sequence Transformer-Transducer Models for Keyword Spotting](https://mlsanity.com/paper/2211.06478/) (2022)

## Built on by (most cited first, 1 of 1)

- [LibriVAD: A Scalable Open Dataset with Deep Learning Benchmarks for Voice Activity Detection](https://mlsanity.com/paper/2512.17281/) (2025)

## Abstract (arXiv)

Keyword spotting systems often struggle to generalize to a diverse population with various accents and age groups. To address this challenge, we propose a novel approach that integrates speaker information into keyword spotting using Feature-wise Linear Modulation (FiLM), a recent method for learning from multiple sources of information. We explore both Text-Dependent and Text-Independent speaker recognition systems to extract speaker information, and we experiment on extracting this information from both the input audio and pre-enrolled user audio. We evaluate our systems on a diverse dataset and achieve a substantial improvement in keyword detection accuracy, particularly among underrepresented speaker groups. Moreover, our proposed approach only requires a small 1% increase in the number of parameters, with a minimum impact on latency and computational cost, which makes it a practical solution for real-world applications.