# Least squares binary quantization of neural networks

arXiv: https://arxiv.org/abs/2001.02786 · HTML: https://mlsanity.com/paper/2001.02786/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2020.

## Built on by (most cited first, 21 of 21)

- [Gaussian Pre-Activations in Neural Networks: Myth or Reality?](https://mlsanity.com/paper/2205.12379/) (2022)
- [Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders](https://mlsanity.com/paper/2304.01016/) (2023)
- [PARQ: Piecewise-Affine Regularized Quantization](https://mlsanity.com/paper/2503.15748/) (2025)
- [Self-Distribution Binary Neural Networks](https://mlsanity.com/paper/2103.02394/) (2021)
- [TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNT](https://mlsanity.com/paper/2104.02233/) (2021)
- [Optimal Brain Apoptosis](https://mlsanity.com/paper/2502.17941/) (2025)
- [FAT: An In-Memory Accelerator with Fast Addition for Ternary Weight Neural Networks](https://mlsanity.com/paper/2201.07634/) (2022)
- [CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method](https://mlsanity.com/paper/2110.02550/) (2021)
- [Signed Binary Weight Networks](https://mlsanity.com/paper/2211.13838/) (2022)
- [Ultra-low Precision Multiplication-free Training for Deep Neural Networks](https://mlsanity.com/paper/2302.14458/) (2023)
- [oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes](https://mlsanity.com/paper/2303.17612/) (2023)
- [Subgraph Stationary Hardware-Software Inference Co-Design](https://mlsanity.com/paper/2306.17266/) (2023)
- [NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models](https://mlsanity.com/paper/2602.06694/) (2026)
- [BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers](https://mlsanity.com/paper/2603.09582/) (2026)
- [On the Acceleration of Deep Neural Network Inference using Quantized Compressed Sensing](https://mlsanity.com/paper/2108.10101/) (2021)
- [IR2Net: Information Restriction and Information Recovery for Accurate Binary Neural Networks](https://mlsanity.com/paper/2210.02637/) (2022)
- [PLUM: Improving Inference Efficiency By Leveraging Repetition-Sparsity Trade-Off](https://mlsanity.com/paper/2312.01581/) (2023)
- [Quantization through Piecewise-Affine Regularization: Optimization and Statistical Guarantees](https://mlsanity.com/paper/2508.11112/) (2025)
- [From Arithmetic to Logic: The Resilience of Logic and Lookup-Based Neural Networks Under Parameter Bit-Flips](https://mlsanity.com/paper/2603.22770/) (2026)
- [GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs](https://mlsanity.com/paper/2603.25385/) (2026)
- [ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights](https://mlsanity.com/paper/2609.38521/) (2026)

## Abstract (arXiv)

Quantizing weights and activations of deep neural networks results in significant improvement in inference efficiency at the cost of lower accuracy. A source of the accuracy gap between full precision and quantized models is the quantization error. In this work, we focus on the binary quantization, in which values are mapped to -1 and 1. We provide a unified framework to analyze different scaling strategies. Inspired by the pareto-optimality of 2-bits versus 1-bit quantization, we introduce a novel 2-bits quantization with provably least squares error. Our quantization algorithms can be implemented efficiently on the hardware using bitwise operations. We present proofs to show that our proposed methods are optimal, and also provide empirical error analysis. We conduct experiments on the ImageNet dataset and show a reduced accuracy gap when using the proposed least squares quantization algorithms.