# ShiftAddNet: A Hardware-Inspired Deep Network

arXiv: https://arxiv.org/abs/2010.12785 · HTML: https://mlsanity.com/paper/2010.12785/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 30 of 52)

- [HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark](https://mlsanity.com/paper/2103.10584/) (2021)
- [Token Shift Transformer for Video Classification](https://mlsanity.com/paper/2108.02432/) (2021)
- [Manifold Regularized Dynamic Network Pruning](https://mlsanity.com/paper/2103.05861/) (2021)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks](https://mlsanity.com/paper/2203.05025/) (2022)
- [CPT: Efficient Deep Neural Network Training via Cyclic Precision](https://mlsanity.com/paper/2101.09868/) (2021)
- ["BNN - BN = ?": Training Binary Neural Networks without Batch Normalization](https://mlsanity.com/paper/2104.08215/) (2021)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)
- [Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators](https://mlsanity.com/paper/2106.06575/) (2021)
- [AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence](https://mlsanity.com/paper/2101.10015/) (2021)
- [GhostSR: Learning Ghost Features for Efficient Image Super-Resolution](https://mlsanity.com/paper/2101.08525/) (2021)
- [ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks](https://mlsanity.com/paper/2205.08119/) (2022)
- [S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks](https://mlsanity.com/paper/2107.03453/) (2021)
- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [PECAN: A Product-Quantized Content Addressable Memory Network](https://mlsanity.com/paper/2208.13571/) (2022)
- [Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model](https://mlsanity.com/paper/2409.15298/) (2024)
- [DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization](https://mlsanity.com/paper/2507.12933/) (2025)
- [An Empirical Study of Adder Neural Networks for Object Detection](https://mlsanity.com/paper/2112.13608/) (2021)
- [DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization](https://mlsanity.com/paper/2208.09708/) (2022)
- [NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks](https://mlsanity.com/paper/2210.13361/) (2022)
- [Multiplierless MP-Kernel Machine For Energy-efficient Edge Devices](https://mlsanity.com/paper/2106.01958/) (2021)
- [RCNet: Reverse Feature Pyramid and Cross-scale Shift Network for Object Detection](https://mlsanity.com/paper/2110.12130/) (2021)
- [Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNs](https://mlsanity.com/paper/2302.03523/) (2022)
- [PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks](https://mlsanity.com/paper/2404.00103/) (2024)
- [Redistribution of Weights and Activations for AdderNet Quantization](https://mlsanity.com/paper/2212.10200/) (2022)
- [Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping](https://mlsanity.com/paper/2308.07641/) (2023)
- [PoTPTQ: A Two-step Power-of-Two Post-training for LLMs](https://mlsanity.com/paper/2507.11959/) (2025)
- [Trimming Down Large Spiking Vision Transformers via Heterogeneous Quantization Search](https://mlsanity.com/paper/2412.05505/) (2024)

## Abstract (arXiv)

Multiplication (e.g., convolution) is arguably a cornerstone of modern deep neural networks (DNNs). However, intensive multiplications cause expensive resource costs that challenge DNNs' deployment on resource-constrained edge devices, driving several attempts for multiplication-less deep networks. This paper presented ShiftAddNet, whose main inspiration is drawn from a common practice in energy-efficient hardware implementation, that is, multiplication can be instead performed with additions and logical bit-shifts. We leverage this idea to explicitly parameterize deep networks in this way, yielding a new type of deep network that involves only bit-shift and additive weight layers. This hardware-inspired ShiftAddNet immediately leads to both energy-efficient inference and training, without compromising the expressive capacity compared to standard DNNs. The two complementary operation types (bit-shift and add) additionally enable finer-grained control of the model's learning capacity, leading to more flexible trade-off between accuracy and (training) efficiency, as well as improved robustness to quantization and pruning. We conduct extensive experiments and ablation studies, all backed up by our FPGA-based ShiftAddNet implementation and energy measurements. Compared to existing DNNs or other multiplication-less models, ShiftAddNet aggressively reduces over 80% hardware-quantified energy cost of DNNs training and inference, while offering comparable or better accuracies. Codes and pre-trained models are available at https://github.com/RICE-EIC/ShiftAddNet.