# Neural Networks with Few Multiplications

arXiv: https://arxiv.org/abs/1510.03009 · HTML: https://mlsanity.com/paper/1510.03009/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2015.

## Built on by (most cited first, 30 of 48)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey](https://mlsanity.com/paper/2203.08737/) (2022)
- [Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning](https://mlsanity.com/paper/2501.03035/) (2025)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization](https://mlsanity.com/paper/2203.11239/) (2022)
- [Distance-aware Quantization](https://mlsanity.com/paper/2108.06983/) (2021)
- [On-device Training: A First Overview on Existing Systems](https://mlsanity.com/paper/2212.00824/) (2022)
- [EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs](https://mlsanity.com/paper/2403.02775/) (2024)
- [PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation](https://mlsanity.com/paper/2403.09192/) (2024)
- [Binary Complex Neural Network Acceleration on FPGA](https://mlsanity.com/paper/2108.04811/) (2021)
- [SPIQ: Data-Free Per-Channel Static Input Quantization](https://mlsanity.com/paper/2203.14642/) (2022)
- [Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction](https://mlsanity.com/paper/2212.05735/) (2022)
- [MKQ-BERT: Quantized BERT with 4-bits Weights and Activations](https://mlsanity.com/paper/2203.13483/) (2022)
- [ULEEN: A Novel Architecture for Ultra Low-Energy Edge Neural Networks](https://mlsanity.com/paper/2304.10618/) (2023)
- [S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks](https://mlsanity.com/paper/2107.03453/) (2021)
- [Random matrices in service of ML footprint: ternary random features with no performance loss](https://mlsanity.com/paper/2110.01899/) (2021)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Boosting Binary Masks for Multi-Domain Learning through Affine Transformations](https://mlsanity.com/paper/2103.13894/) (2021)
- [XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networks](https://mlsanity.com/paper/2111.10854/) (2021)
- [Distributed Machine Learning for UAV Swarms: Computing, Sensing, and Semantics](https://mlsanity.com/paper/2301.00912/) (2023)
- [Number Systems for Deep Neural Network Architectures: A Survey](https://mlsanity.com/paper/2307.05035/) (2023)
- [MinUn: Accurate ML Inference on Microcontrollers](https://mlsanity.com/paper/2210.16556/) (2022)
- [ATHEENA: A Toolflow for Hardware Early-Exit Network Automation](https://mlsanity.com/paper/2304.08400/) (2023)
- [The Ecological Footprint of Neural Machine Translation Systems](https://mlsanity.com/paper/2202.02170/) (2022)

## Abstract (arXiv)

For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.