# Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensors

arXiv: https://arxiv.org/abs/2408.12481 · HTML: https://mlsanity.com/paper/2408.12481/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 30 of 34)

- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference](https://mlsanity.com/paper/2204.07305/) (2022)
- [Hello Edge: Keyword Spotting on Microcontrollers](https://mlsanity.com/paper/1711.07128/) (2017)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [TinyOL: TinyML with Online-Learning on Microcontrollers](https://mlsanity.com/paper/2103.08295/) (2021)
- [Streaming keyword spotting on mobile devices](https://mlsanity.com/paper/2005.06720/) (2020)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training](https://mlsanity.com/paper/2011.13439/) (2020)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [Few-Shot Keyword Spotting in Any Language](https://mlsanity.com/paper/2104.01454/) (2021)
- [End-to-End Streaming Keyword Spotting](https://mlsanity.com/paper/1812.02802/) (2018)
- [Learning Efficient Representations for Keyword Spotting with Triplet Loss](https://mlsanity.com/paper/2101.04792/) (2021)
- [Efficient keyword spotting using dilated convolutions and gating](https://mlsanity.com/paper/1811.07684/) (2018)
- [Query-by-example on-device keyword spotting](https://mlsanity.com/paper/1910.05171/) (2019)
- [Few-Shot Keyword Spotting With Prototypical Networks](https://mlsanity.com/paper/2007.14463/) (2020)
- [TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge](https://mlsanity.com/paper/2307.09988/) (2023)
- [On-Device Domain Learning for Keyword Spotting on Low-Power Extreme Edge Embedded Systems](https://mlsanity.com/paper/2403.10549/) (2024)
- [LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms](https://mlsanity.com/paper/2311.11420/) (2023)
- [Learning To Detect Keyword Parts And Whole By Smoothed Max Pooling](https://mlsanity.com/paper/2001.09246/) (2020)
- [Noisy student-teacher training for robust keyword spotting](https://mlsanity.com/paper/2106.01604/) (2021)
- [Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllers](https://mlsanity.com/paper/2305.19167/) (2023)
- [Small-Footprint Open-Vocabulary Keyword Spotting with Quantized LSTM Networks](https://mlsanity.com/paper/2002.10851/) (2020)
- [Metric Learning for Keyword Spotting](https://mlsanity.com/paper/2005.08776/) (2020)
- [Teaching keyword spotters to spot new keywords with limited examples](https://mlsanity.com/paper/2106.02443/) (2021)
- [Few-Shot Open-Set Learning for On-Device Customization of KeyWord Spotting Systems](https://mlsanity.com/paper/2306.02161/) (2023)
- [Dummy Prototypical Networks for Few-Shot Open-Set Keyword Spotting](https://mlsanity.com/paper/2206.13691/) (2022)

## Built on by (most cited first, 7 of 7)

- [Enhancing Few-shot Keyword Spotting Performance through Pre-Trained Self-supervised Speech Models](https://mlsanity.com/paper/2506.17686/) (2025)
- [Multi-modal On-Device Learning for Monocular Depth Estimation on Ultra-low-power MCUs](https://mlsanity.com/paper/2512.00086/) (2025)
- [Domain-Incremental Continual Learning for Robust and Efficient Keyword Spotting in Resource Constrained Systems](https://mlsanity.com/paper/2601.16158/) (2026)
- [OnDA: On-device Channel Pruning for Efficient Personalized Keyword Spotting](https://mlsanity.com/paper/2603.02247/) (2026)
- [Scaling few-shot spoken word classification with generative meta-continual learning](https://mlsanity.com/paper/2605.13075/) (2026)
- [What changes after deployment? A survey on On-device Learning in TinyML](https://mlsanity.com/paper/2605.31226/) (2026)
- [TASTE: Throughput-Aware Batch Size Tuning for On-Device Edge Learning](https://mlsanity.com/paper/2609.07444/) (2026)

## Abstract (arXiv)

This paper proposes a self-learning method to incrementally train (fine-tune) a personalized Keyword Spotting (KWS) model after the deployment on ultra-low power smart audio sensors. We address the fundamental problem of the absence of labeled training data by assigning pseudo-labels to the new recorded audio frames based on a similarity score with respect to few user recordings. By experimenting with multiple KWS models with a number of parameters up to 0.5M on two public datasets, we show an accuracy improvement of up to +19.2% and +16.0% vs. the initial models pretrained on a large set of generic keywords. The labeling task is demonstrated on a sensor system composed of a low-power microphone and an energy-efficient Microcontroller (MCU). By efficiently exploiting the heterogeneous processing engines of the MCU, the always-on labeling task runs in real-time with an average power cost of up to 8.2 mW. On the same platform, we estimate an energy cost for on-device training 10x lower than the labeling energy if sampling a new utterance every 6.1 s or 18.8 s with a DS-CNN-S or a DS-CNN-M model. Our empirical result paves the way to self-adaptive personalized KWS sensors at the extreme edge.