# Sub 8-Bit Quantization of Streaming Keyword Spotting Models for Embedded Chipsets

arXiv: https://arxiv.org/abs/2207.06920 · HTML: https://mlsanity.com/paper/2207.06920/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Accurate Detection of Wake Word Start and End Using a CNN](https://mlsanity.com/paper/2008.03790/) (2020)
- [Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutions](https://mlsanity.com/paper/1911.02086/) (2019)
- [Small-Footprint Keyword Spotting with Multi-Scale Temporal Convolution](https://mlsanity.com/paper/2010.09960/) (2020)
- [On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition](https://mlsanity.com/paper/1603.08042/) (2016)
- [Hardware Aware Training for Efficient Keyword Spotting on General Purpose and Specialized Hardware](https://mlsanity.com/paper/2009.04465/) (2020)
- [On Front-end Gain Invariant Modeling for Wake Word Spotting](https://mlsanity.com/paper/2010.06676/) (2020)
- [Compression of Acoustic Event Detection Models with Low-rank Matrix Factorization and Quantization Training](https://mlsanity.com/paper/1905.00855/) (2019)

## Built on by (most cited first, 2 of 2)

- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Fixed-point quantization aware training for on-device keyword-spotting](https://mlsanity.com/paper/2303.02284/) (2023)

## Abstract (arXiv)

We propose a novel 2-stage sub 8-bit quantization aware training algorithm for all components of a 250K parameter feedforward, streaming, state-free keyword spotting model. For the 1st-stage, we adapt a recently proposed quantization technique using a non-linear transformation with tanh(.) on dense layer weights. In the 2nd-stage, we use linear quantization methods on the rest of the network, including other parameters (bias, gain, batchnorm), inputs, and activations. We conduct large scale experiments, training on 26,000 hours of de-identified production, far-field and near-field audio data (evaluating on 4,000 hours of data). We organize our results in two embedded chipset settings: a) with commodity ARM NEON instruction set and 8-bit containers, we present accuracy, CPU, and memory results using sub 8-bit weights (4, 5, 8-bit) and 8-bit quantization of rest of the network; b) with off-the-shelf neural network accelerators, for a range of weight bit widths (1 and 5-bit), while presenting accuracy results, we project reduction in memory utilization. In both configurations, our results show that the proposed algorithm can achieve: a) parity with a full floating point model's operating point on a detection error tradeoff (DET) curve in terms of false detection rate (FDR) at false rejection rate (FRR); b) significant reduction in compute and memory, yielding up to 3 times improvement in CPU consumption and more than 4 times improvement in memory consumption.