# Training Binary Neural Networks using the Bayesian Learning Rule

arXiv: https://arxiv.org/abs/2002.10778 · HTML: https://mlsanity.com/paper/2002.10778/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2020.

## Built on by (most cited first, 23 of 23)

- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [SAM as an Optimal Relaxation of Bayes](https://mlsanity.com/paper/2210.01620/) (2022)
- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [Bayesian Continual Learning via Spiking Neural Networks](https://mlsanity.com/paper/2208.13723/) (2022)
- [Low-Precision Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2206.09909/) (2022)
- [AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks](https://mlsanity.com/paper/2112.02880/) (2021)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Quantum HyperNetworks: Training Binary Neural Networks in Quantum Superposition](https://mlsanity.com/paper/2301.08292/) (2023)
- [Supplementing Recurrent Neural Networks with Annealing to Solve Combinatorial Optimization Problems](https://mlsanity.com/paper/2207.08189/) (2022)
- [BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule](https://mlsanity.com/paper/2111.13204/) (2021)
- [Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators](https://mlsanity.com/paper/2110.03549/) (2021)
- [Seeking Interpretability and Explainability in Binary Activated Neural Networks](https://mlsanity.com/paper/2209.03450/) (2022)
- [The Lie-Group Bayesian Learning Rule](https://mlsanity.com/paper/2303.04397/) (2023)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Effective Neural Network L0 Regularization With BinMask](https://mlsanity.com/paper/2304.11237/) (2023)
- [Probabilistic Approach to Black-Box Binary Optimization with Budget Constraints: Application to Sensor Placement](https://mlsanity.com/paper/2406.05830/) (2024)
- [Quantized Proximal Averaging Network for Analysis Sparse Coding](https://mlsanity.com/paper/2105.06211/) (2021)
- [Understanding weight-magnitude hyperparameters in training binary networks](https://mlsanity.com/paper/2303.02452/) (2023)
- [Long-Tailed Recognition on Binary Networks by Calibrating A Pre-trained Model](https://mlsanity.com/paper/2404.00285/) (2024)
- [Bayes2IMC: In-Memory Computing for Bayesian Binary Neural Networks](https://mlsanity.com/paper/2411.07902/) (2024)
- [A Principled Bayesian Framework for Training Binary and Spiking Neural Networks](https://mlsanity.com/paper/2505.17962/) (2025)
- [SVRG and Beyond via Posterior Correction](https://mlsanity.com/paper/2512.01930/) (2025)
- [Active Continual Learning with Metaplastic Binary Bayesian Neural Networks](https://mlsanity.com/paper/2605.30198/) (2026)

## Abstract (arXiv)

Neural networks with binary weights are computation-efficient and hardware-friendly, but their training is challenging because it involves a discrete optimization problem. Surprisingly, ignoring the discrete nature of the problem and using gradient-based methods, such as the Straight-Through Estimator, still works well in practice. This raises the question: are there principled approaches which justify such methods? In this paper, we propose such an approach using the Bayesian learning rule. The rule, when applied to estimate a Bernoulli distribution over the binary weights, results in an algorithm which justifies some of the algorithmic choices made by the previous approaches. The algorithm not only obtains state-of-the-art performance, but also enables uncertainty estimation for continual learning to avoid catastrophic forgetting. Our work provides a principled approach for training binary neural networks which justifies and extends existing approaches.