# Low-bit Shift Network for End-to-End Spoken Language Understanding

arXiv: https://arxiv.org/abs/2207.07497 · HTML: https://mlsanity.com/paper/2207.07497/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 15 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Benchmarking TinyML Systems: Challenges and Direction](https://mlsanity.com/paper/2003.04821/) (2020)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [Spoken Language Understanding on the Edge](https://mlsanity.com/paper/1810.12735/) (2018)
- [Exploring Transfer Learning For End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2012.08549/) (2020)
- [ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks](https://mlsanity.com/paper/1706.02393/) (2017)
- [A Simplified Fully Quantized Transformer for End-to-end Speech Recognition](https://mlsanity.com/paper/1911.03604/) (2019)
- [S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks](https://mlsanity.com/paper/2107.03453/) (2021)

## Built on by (most cited first, 1 of 1)

- [DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization](https://mlsanity.com/paper/2208.09708/) (2022)

## Abstract (arXiv)

Deep neural networks (DNN) have achieved impressive success in multiple domains. Over the years, the accuracy of these models has increased with the proliferation of deeper and more complex architectures. Thus, state-of-the-art solutions are often computationally expensive, which makes them unfit to be deployed on edge computing platforms. In order to mitigate the high computation, memory, and power requirements of inferring convolutional neural networks (CNNs), we propose the use of power-of-two quantization, which quantizes continuous parameters into low-bit power-of-two values. This reduces computational complexity by removing expensive multiplication operations and with the use of low-bit weights. ResNet is adopted as the building block of our solution and the proposed model is evaluated on a spoken language understanding (SLU) task. Experimental results show improved performance for shift neural network architectures, with our low-bit quantization achieving 98.76 % on the test set which is comparable performance to its full-precision counterpart and state-of-the-art solutions.