# Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization

arXiv: https://arxiv.org/abs/1906.02107 · HTML: https://mlsanity.com/paper/1906.02107/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2019.

## Built on by (most cited first, 30 of 60)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://mlsanity.com/paper/2402.04291/) (2024)
- [Overcoming Oscillations in Quantization-Aware Training](https://mlsanity.com/paper/2203.11086/) (2022)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [PB-LLM: Partially Binarized Large Language Models](https://mlsanity.com/paper/2310.00034/) (2023)
- [PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression](https://mlsanity.com/paper/2405.14852/) (2024)
- [Oscillation-free Quantization for Low-bit Vision Transformers](https://mlsanity.com/paper/2302.02210/) (2023)
- [ReCU: Reviving the Dead Weights in Binary Neural Networks](https://mlsanity.com/paper/2103.12369/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [Learning Frequency Domain Approximation for Binary Neural Networks](https://mlsanity.com/paper/2103.00841/) (2021)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training](https://mlsanity.com/paper/2308.06689/) (2023)
- [SiMaN: Sign-to-Magnitude Network Binarization](https://mlsanity.com/paper/2102.07981/) (2021)
- ["BNN - BN = ?": Training Binary Neural Networks without Batch Normalization](https://mlsanity.com/paper/2104.08215/) (2021)
- [Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket](https://mlsanity.com/paper/2211.12933/) (2022)
- [Training Dynamical Binary Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2103.08953/) (2021)
- [Enabling Binary Neural Network Training on the Edge](https://mlsanity.com/paper/2102.04270/) (2021)
- [Demystifying and Generalizing BinaryConnect](https://mlsanity.com/paper/2110.13220/) (2021)
- [AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks](https://mlsanity.com/paper/2112.02880/) (2021)
- [TerViT: An Efficient Ternary Vision Transformer](https://mlsanity.com/paper/2201.08050/) (2022)
- [Equal Bits: Enforcing Equally Distributed Binary Network Weights](https://mlsanity.com/paper/2112.03406/) (2021)
- [Globally Optimal Training of Neural Networks with Threshold Activation Functions](https://mlsanity.com/paper/2303.03382/) (2023)
- [BoolNet: Minimizing The Energy Consumption of Binary Neural Networks](https://mlsanity.com/paper/2106.06991/) (2021)
- [HideNseek: Federated Lottery Ticket via Server-side Pruning and Sign Supermask](https://mlsanity.com/paper/2206.04385/) (2022)
- [Custom Gradient Estimators are Straight-Through Estimators in Disguise](https://mlsanity.com/paper/2405.05171/) (2024)
- [Towards Accurate Binary Spiking Neural Networks: Learning with Adaptive Gradient Modulation Mechanism](https://mlsanity.com/paper/2502.14344/) (2025)
- [Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models](https://mlsanity.com/paper/2509.18763/) (2025)
- [Neural-based classification rule learning for sequential data](https://mlsanity.com/paper/2302.11286/) (2023)
- [Enabling On-device Continual Learning with Binary Neural Networks](https://mlsanity.com/paper/2401.09916/) (2024)

## Abstract (arXiv)

Optimization of Binarized Neural Networks (BNNs) currently relies on real-valued latent weights to accumulate small update steps. In this paper, we argue that these latent weights cannot be treated analogously to weights in real-valued networks. Instead their main role is to provide inertia during training. We interpret current methods in terms of inertia and provide novel insights into the optimization of BNNs. We subsequently introduce the first optimizer specifically designed for BNNs, Binary Optimizer (Bop), and demonstrate its performance on CIFAR-10 and ImageNet. Together, the redefinition of latent weights as inertia and the introduction of Bop enable a better understanding of BNN optimization and open up the way for further improvements in training methodologies for BNNs. Code is available at: https://github.com/plumerai/rethinking-bnn-optimization