# Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

arXiv: https://arxiv.org/abs/1602.02830 · HTML: https://mlsanity.com/paper/1602.02830/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2016.

## Built on by (most cited first, 30 of 105)

- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [RPTQ: Reorder-based Post-training Quantization for Large Language Models](https://mlsanity.com/paper/2304.01089/) (2023)
- [The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models](https://mlsanity.com/paper/2203.07259/) (2022)
- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks](https://mlsanity.com/paper/2106.12379/) (2021)
- [Differentiable Model Compression via Pseudo Quantization Noise](https://mlsanity.com/paper/2104.09987/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Content-Aware GAN Compression](https://mlsanity.com/paper/2104.02244/) (2021)
- [SDQ: Stochastic Differentiable Quantization with Mixed Precision](https://mlsanity.com/paper/2206.04459/) (2022)
- [Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning](https://mlsanity.com/paper/2109.01905/) (2021)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks](https://mlsanity.com/paper/2109.12872/) (2021)
- [Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution](https://mlsanity.com/paper/2106.02295/) (2021)
- [CPT: Efficient Deep Neural Network Training via Cyclic Precision](https://mlsanity.com/paper/2101.09868/) (2021)
- ["BNN - BN = ?": Training Binary Neural Networks without Batch Normalization](https://mlsanity.com/paper/2104.08215/) (2021)
- [Lite Pose: Efficient Architecture Design for 2D Human Pose Estimation](https://mlsanity.com/paper/2205.01271/) (2022)
- [S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration](https://mlsanity.com/paper/2102.08946/) (2021)
- [Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge](https://mlsanity.com/paper/2301.10281/) (2023)
- [Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descent](https://mlsanity.com/paper/2102.07567/) (2021)
- [Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks](https://mlsanity.com/paper/2104.07085/) (2021)
- [Dynamic Binary Neural Network by learning channel-wise thresholds](https://mlsanity.com/paper/2110.05185/) (2021)
- [Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networks](https://mlsanity.com/paper/2201.02711/) (2022)
- [Weightless Neural Networks for Efficient Edge Inference](https://mlsanity.com/paper/2203.01479/) (2022)
- [IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectors](https://mlsanity.com/paper/2210.03477/) (2022)
- [Leaky Nets: Recovering Embedded Neural Network Models and Inputs through Simple Power and Timing Side-Channels -- Attacks and Defenses](https://mlsanity.com/paper/2103.14739/) (2021)

## Abstract (arXiv)

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs we conduct two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.