# Small-footprint slimmable networks for keyword spotting

arXiv: https://arxiv.org/abs/2304.12183 · HTML: https://mlsanity.com/paper/2304.12183/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [Keyword Transformer: A Self-Attention Model for Keyword Spotting](https://mlsanity.com/paper/2104.00769/) (2021)
- [Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution](https://mlsanity.com/paper/1701.00299/) (2017)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [Dynamic Sparsity Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2005.10627/) (2020)
- [Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernet](https://mlsanity.com/paper/2110.08352/) (2021)
- [Tiny-CRNN: Streaming Wakeword Detection In A Low Footprint Setting](https://mlsanity.com/paper/2109.14725/) (2021)

## Built on by (most cited first, 1 of 1)

- [Keyword Spotting with Hyper-Matched Filters for Small Footprint Devices](https://mlsanity.com/paper/2508.04857/) (2025)

## Abstract (arXiv)

In this work, we present Slimmable Neural Networks applied to the problem of small-footprint keyword spotting. We show that slimmable neural networks allow us to create super-nets from Convolutioanl Neural Networks and Transformers, from which sub-networks of different sizes can be extracted. We demonstrate the usefulness of these models on in-house Alexa data and Google Speech Commands, and focus our efforts on models for the on-device use case, limiting ourselves to less than 250k parameters. We show that slimmable models can match (and in some cases, outperform) models trained from scratch. Slimmable neural networks are therefore a class of models particularly useful when the same functionality is to be replicated at different memory and compute budgets, with different accuracy requirements.