# AdderNet: Do We Really Need Multiplications in Deep Learning?

arXiv: https://arxiv.org/abs/1912.13200 · HTML: https://mlsanity.com/paper/1912.13200/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2019.

## Built on by (most cited first, 30 of 82)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation](https://mlsanity.com/paper/2204.05525/) (2022)
- [Rewrite the Stars](https://mlsanity.com/paper/2403.19967/) (2024)
- [Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation](https://mlsanity.com/paper/2210.04524/) (2022)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons](https://mlsanity.com/paper/2102.05363/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [Revisiting Dynamic Convolution via Matrix Decomposition](https://mlsanity.com/paper/2103.08756/) (2021)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [Differentiable Weightless Neural Networks](https://mlsanity.com/paper/2410.11112/) (2024)
- [MicroNet: Improving Image Recognition with Extremely Low FLOPs](https://mlsanity.com/paper/2108.05894/) (2021)
- [Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks](https://mlsanity.com/paper/2109.12872/) (2021)
- [Scalable MatMul-free Language Modeling](https://mlsanity.com/paper/2406.02528/) (2024)
- [EcoFormer: Energy-Saving Attention with Linear Complexity](https://mlsanity.com/paper/2209.09004/) (2022)
- [DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks](https://mlsanity.com/paper/2206.00843/) (2022)
- [GhostNets on Heterogeneous Devices via Cheap Operations](https://mlsanity.com/paper/2201.03297/) (2022)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)
- [MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations](https://mlsanity.com/paper/2105.07085/) (2021)
- [Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory](https://mlsanity.com/paper/2310.06756/) (2023)
- [MoCoViT: Mobile Convolutional Vision Transformer](https://mlsanity.com/paper/2205.12635/) (2022)
- [Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables](https://mlsanity.com/paper/2303.14506/) (2023)
- [AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence](https://mlsanity.com/paper/2101.10015/) (2021)
- [Min-Max-Plus Neural Networks](https://mlsanity.com/paper/2102.06358/) (2021)
- [ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks](https://mlsanity.com/paper/2205.08119/) (2022)
- [S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks](https://mlsanity.com/paper/2107.03453/) (2021)
- [PECAN: A Product-Quantized Content Addressable Memory Network](https://mlsanity.com/paper/2208.13571/) (2022)
- [PQA: Exploring the Potential of Product Quantization in DNN Hardware Acceleration](https://mlsanity.com/paper/2305.18334/) (2023)

## Abstract (arXiv)

Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-correlation to measure the similarity between input feature and convolution filters, which involves massive multiplications between float values. In this paper, we present adder networks (AdderNets) to trade these massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the ℓ1-norm distance between filters and input feature as the output response. The influence of this new similarity measure on the optimization of neural network have been thoroughly analyzed. To achieve a better performance, we develop a special back-propagation approach for AdderNets by investigating the full-precision gradient. We then propose an adaptive learning rate strategy to enhance the training procedure of AdderNets according to the magnitude of each neuron's gradient. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in convolution layer. The codes are publicly available at: https://github.com/huaweinoah/AdderNet.