# Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statistics

arXiv: https://arxiv.org/abs/2307.12660 · HTML: https://mlsanity.com/paper/2307.12660/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 26 of 26)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2211.13131/) (2022)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis](https://mlsanity.com/paper/1909.01520/) (2019)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Deep Residual Learning for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1710.10361/) (2017)
- [Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization](https://mlsanity.com/paper/1712.01034/) (2017)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)
- [Stochastic Pooling for Regularization of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1301.3557/) (2013)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Continual Learning at the Edge: Real-Time Training on Smartphone Devices](https://mlsanity.com/paper/2105.13127/) (2021)
- [Online Continual Learning of End-to-End Speech Recognition Models](https://mlsanity.com/paper/2207.05071/) (2022)
- [F-SIOL-310: A Robotic Dataset and Benchmark for Few-Shot Incremental Object Learning](https://mlsanity.com/paper/2103.12242/) (2021)
- [Progressive Continual Learning for Spoken Keyword Spotting](https://mlsanity.com/paper/2201.12546/) (2022)
- [Rainbow Keywords: Efficient Incremental Learning for Online Spoken Keyword Spotting](https://mlsanity.com/paper/2203.16361/) (2022)
- [Multi-Task Learning with High-Order Statistics for X-vector based Text-Independent Speaker Verification](https://mlsanity.com/paper/1903.12058/) (2019)
- [Study on the temporal pooling used in deep neural networks for speaker verification](https://mlsanity.com/paper/2105.04310/) (2021)
- [On the challenges to learn from Natural Data Streams](https://mlsanity.com/paper/2301.03495/) (2023)

## Built on by (most cited first, 9 of 9)

- [Recent Advances of Foundation Language Models-based Continual Learning: A Survey](https://mlsanity.com/paper/2405.18653/) (2024)
- [Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems](https://mlsanity.com/paper/2407.03645/) (2024)
- [HOP to the Next Tasks and Domains for Continual Learning in NLP](https://mlsanity.com/paper/2402.18449/) (2024)
- [Object-conditioned Bag of Instances for Few-Shot Personalized Instance Recognition](https://mlsanity.com/paper/2404.01397/) (2024)
- [Efficient Continual Learning in Keyword Spotting using Binary Neural Networks](https://mlsanity.com/paper/2505.02469/) (2025)
- [Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR](https://mlsanity.com/paper/2506.21577/) (2025)
- [Adaptive Noise Resilient Keyword Spotting Using One-Shot Learning](https://mlsanity.com/paper/2505.09304/) (2025)
- [MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devices](https://mlsanity.com/paper/2504.00174/) (2025)
- [Domain-Incremental Continual Learning for Robust and Efficient Keyword Spotting in Resource Constrained Systems](https://mlsanity.com/paper/2601.16158/) (2026)

## Abstract (arXiv)

Keyword Spotting (KWS) models on embedded devices should adapt fast to new user-defined words without forgetting previous ones. Embedded devices have limited storage and computational resources, thus, they cannot save samples or update large models. We consider the setup of embedded online continual learning (EOCL), where KWS models with frozen backbone are trained to incrementally recognize new words from a non-repeated stream of samples, seen one at a time. To this end, we propose Temporal Aware Pooling (TAP) which constructs an enriched feature space computing high-order moments of speech features extracted by a pre-trained backbone. Our method, TAP-SLDA, updates a Gaussian model for each class on the enriched feature space to effectively use audio representations. In experimental analyses, TAP-SLDA outperforms competitors on several setups, backbones, and baselines, bringing a relative average gain of 11.3% on the GSC dataset.