# Neural Network Quantization by Learning Low-Loss Subspaces

arXiv: https://arxiv.org/abs/2606.25087 · HTML: https://mlsanity.com/paper/2606.25087/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 54 of its 69 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 30 of 54)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://mlsanity.com/paper/1903.05662/) (2019)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Residual Feature Distillation Network for Lightweight Image Super-Resolution](https://mlsanity.com/paper/2009.11551/) (2020)
- [Overcoming Oscillations in Quantization-Aware Training](https://mlsanity.com/paper/2203.11086/) (2022)
- [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016)
- [Revisiting Catastrophic Forgetting in Large Language Model Tuning](https://mlsanity.com/paper/2406.04836/) (2024)
- [Loss Aware Post-training Quantization](https://mlsanity.com/paper/1911.07190/) (2019)
- [Network Quantization with Element-wise Gradient Scaling](https://mlsanity.com/paper/2104.00903/) (2021)
- [Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes](https://mlsanity.com/paper/1706.10239/) (2017)
- [Learning Neural Network Subspaces](https://mlsanity.com/paper/2102.10472/) (2021)
- [Robust Quantization: One Model to Rule Them All](https://mlsanity.com/paper/2002.07686/) (2020)
- [Bridging Discrete and Backpropagation: Straight-Through and Beyond](https://mlsanity.com/paper/2304.08612/) (2023)

## Abstract (arXiv)

Neural network quantization aims to find a discrete representation of parameters that preserves the performance of a full-precision (FP) model as faithfully as possible. Enforcing discrete constraints perturbs parameters away from a well-optimized minimum, generally resulting in performance degradation. Recent studies indicate that low-loss FP solutions are not isolated, but instead belong to connected low-loss subspaces of the loss landscape, where the loss maintains nearly the same minimum value. Models sampled from these subspaces are diverse and retain high accuracy. This raises the question: can a quantized model be constructed to lie within a low-loss subspace of the FP model, thereby automatically preserving performance? We address this question by learning quantization-aware linear paths in weight space optimized to minimize loss. We demonstrate that the midpoint of the resulting subspace is, by design, quantization-friendly and that its direct quantization yields performance comparable to that of quantization-aware training. The proposed procedure offers a novel perspective on weight quantization and, in contrast to conventional methods, neither relies on the straight-through estimator nor involves explicit discretization during training.