# ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation

arXiv: https://arxiv.org/abs/2407.02881 · HTML: https://mlsanity.com/paper/2407.02881/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation](https://mlsanity.com/paper/1901.02985/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions](https://mlsanity.com/paper/1711.08141/) (2017)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [AdderNet: Do We Really Need Multiplications in Deep Learning?](https://mlsanity.com/paper/1912.13200/) (2019)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1903.05285/) (2019)
- [AdderSR: Towards Energy Efficient Image Super-Resolution](https://mlsanity.com/paper/2009.08891/) (2020)
- [NAAS: Neural Accelerator Architecture Search](https://mlsanity.com/paper/2105.13258/) (2021)
- [Network Augmentation for Tiny Deep Learning](https://mlsanity.com/paper/2110.08890/) (2021)
- [DNN-Chip Predictor: An Analytical Performance Predictor for DNN Accelerators with Various Dataflows and Hardware Architectures](https://mlsanity.com/paper/2002.11270/) (2020)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)
- [AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence](https://mlsanity.com/paper/2101.10015/) (2021)
- [ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks](https://mlsanity.com/paper/2205.08119/) (2022)
- [NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks](https://mlsanity.com/paper/2210.13361/) (2022)

## Built on by (most cited first, 1 of 1)

- [QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization](https://mlsanity.com/paper/2609.00224/) (2026)

## Abstract (arXiv)

Operators devoid of multiplication, such as Shift and Add, have gained prominence for their compatibility with hardware. However, neural networks (NNs) employing these operators typically exhibit lower accuracy compared to conventional NNs with identical structures. ShiftAddAug uses costly multiplication to augment efficient but less powerful multiplication-free operators, improving performance without any inference overhead. It puts a ShiftAdd tiny NN into a large multiplicative model and encourages it to be trained as a sub-model to obtain additional supervision. In order to solve the weight discrepancy problem between hybrid operators, a new weight sharing method is proposed. Additionally, a novel two stage neural architecture search is used to obtain better augmentation effects for smaller but stronger multiplication-free tiny neural networks. The superiority of ShiftAddAug is validated through experiments in image classification and semantic segmentation, consistently delivering noteworthy enhancements. Remarkably, it secures up to a 4.95% increase in accuracy on the CIFAR100 compared to its directly trained counterparts, even surpassing the performance of multiplicative NNs.