# DeepShift: Towards Multiplication-Less Neural Networks

arXiv: https://arxiv.org/abs/1905.13298 · HTML: https://mlsanity.com/paper/1905.13298/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 30 of 55)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Rewrite the Stars](https://mlsanity.com/paper/2403.19967/) (2024)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [Differentiable Weightless Neural Networks](https://mlsanity.com/paper/2410.11112/) (2024)
- [Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks](https://mlsanity.com/paper/2203.05025/) (2022)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)
- [AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence](https://mlsanity.com/paper/2101.10015/) (2021)
- [ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks](https://mlsanity.com/paper/2205.08119/) (2022)
- [S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks](https://mlsanity.com/paper/2107.03453/) (2021)
- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization](https://mlsanity.com/paper/2208.09708/) (2022)
- [PECAN: A Product-Quantized Content Addressable Memory Network](https://mlsanity.com/paper/2208.13571/) (2022)
- [DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization](https://mlsanity.com/paper/2507.12933/) (2025)
- [WARP-LUTs -- Walsh-Assisted Relaxation for Probabilistic Look Up Tables](https://mlsanity.com/paper/2510.15655/) (2025)
- [NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks](https://mlsanity.com/paper/2210.13361/) (2022)
- [Multiplierless MP-Kernel Machine For Energy-efficient Edge Devices](https://mlsanity.com/paper/2106.01958/) (2021)
- [Energy Efficient Hardware Acceleration of Neural Networks with Power-of-Two Quantisation](https://mlsanity.com/paper/2209.15257/) (2022)
- [Learning Discretized Neural Networks under Ricci Flow](https://mlsanity.com/paper/2302.03390/) (2023)
- [P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer](https://mlsanity.com/paper/2405.19915/) (2024)
- [PoTPTQ: A Two-step Power-of-Two Post-training for LLMs](https://mlsanity.com/paper/2507.11959/) (2025)
- [Towards Leveraging AutoML for Sustainable Deep Learning: A Multi-Objective HPO Approach on Deep Shift Neural Networks](https://mlsanity.com/paper/2404.01965/) (2024)
- [PowerYOLO: Mixed Precision Model for Hardware Efficient Object Detection with Event Data](https://mlsanity.com/paper/2407.08272/) (2024)
- [Deep Lookup Network](https://mlsanity.com/paper/2509.13662/) (2025)
- [WARP Logic Neural Networks](https://mlsanity.com/paper/2602.03527/) (2026)
- [CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method](https://mlsanity.com/paper/2110.02550/) (2021)
- [Computational Complexity Evaluation of Neural Network Applications in Signal Processing](https://mlsanity.com/paper/2206.12191/) (2022)
- [Low-bit Shift Network for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2207.07497/) (2022)
- [Ultra-low Precision Multiplication-free Training for Deep Neural Networks](https://mlsanity.com/paper/2302.14458/) (2023)
- [Attention, Distillation, and Tabularization: Towards Practical Neural Network-Based Prefetching](https://mlsanity.com/paper/2401.06362/) (2023)

## Abstract (arXiv)

The high computation, memory, and power budgets of inferring convolutional neural networks (CNNs) are major bottlenecks of model deployment to edge computing platforms, e.g., mobile devices and IoT. Moreover, training CNNs is time and energy-intensive even on high-grade servers. Convolution layers and fully connected layers, because of their intense use of multiplications, are the dominant contributor to this computation budget. We propose to alleviate this problem by introducing two new operations: convolutional shifts and fully-connected shifts which replace multiplications with bitwise shift and sign flipping during both training and inference. During inference, both approaches require only 5 bits (or less) to represent the weights. This family of neural network architectures (that use convolutional shifts and fully connected shifts) is referred to as DeepShift models. We propose two methods to train DeepShift models: DeepShift-Q which trains regular weights constrained to powers of 2, and DeepShift-PS that trains the values of the shifts and sign flips directly. Very close accuracy, and in some cases higher accuracy, to baselines are achieved. Converting pre-trained 32-bit floating-point baseline models of ResNet18, ResNet50, VGG16, and GoogleNet to DeepShift and training them for 15 to 30 epochs, resulted in Top-1/Top-5 accuracies higher than that of the original model. Last but not least, we implemented the convolutional shifts and fully connected shift GPU kernels and showed a reduction in latency time of 25% when inferring ResNet18 compared to unoptimized multiplication-based GPU kernels. The code can be found at https://github.com/mostafaelhoushi/DeepShift.