# Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models

arXiv: https://arxiv.org/abs/2607.14630 · HTML: https://mlsanity.com/paper/2607.14630/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 24 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 22 of 22)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [ZeroQ: A Novel Zero Shot Quantization Framework](https://mlsanity.com/paper/2001.00281/) (2020)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://mlsanity.com/paper/2402.04291/) (2024)
- [OneBit: Towards Extremely Low-bit Large Language Models](https://mlsanity.com/paper/2402.11295/) (2024)
- [ProxQuant: Quantized Neural Networks via Proximal Operators](https://mlsanity.com/paper/1810.00861/) (2018)
- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [Model-Preserving Adaptive Rounding](https://mlsanity.com/paper/2505.22988/) (2025)
- [Mirror Descent View for Neural Network Quantization](https://mlsanity.com/paper/1910.08237/) (2019)
- [Rethinking Residual Errors in Compensation-based LLM Quantization](https://mlsanity.com/paper/2604.07955/) (2026)
- [Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data](https://mlsanity.com/paper/2302.10899/) (2023)
- [TurboBoA: Faster and Exact Attention-aware Quantization without Backpropagation](https://mlsanity.com/paper/2602.04929/) (2026)

## Abstract (arXiv)

Layer-wise post-training quantization of large language models minimizes each layer's reconstruction error in isolation, allowing quantization errors to accumulate across depth and causing severe degradation in extreme low-bit regimes. We formulate quantization as a joint optimization over the discrete codes and scales of all layers, driven by two mechanisms: (i) cross-layer error compensation, which maintains the network-level accumulated error through the recursion e_{l+1} = A_l e_l + q_l, with a propagation operator A_l derived from the layer's input differential and a local quantization residual q_l evaluated at teacher features; and (ii) finite-sample feature-statistics matching, which aligns means, projected covariances, and centered empirical kernels between the full-precision and quantized networks under relative normalization. We prove that instantiating the propagation operator as a finite difference of the quantized network makes the recursion exact for arbitrary nonlinear layers, enabling an efficient forward-difference implementation. Binary weights are optimized via a mirror-descent parameterization u = tanh(beta*z) with annealed inverse temperature and group-wise log-scales. On Qwen2.5-1.5B with 1.125-bit group-binary weights, error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher, outperforming logit distillation (14.09 +/- 0.53; 32 percent relative, more than 8 sigma over 3 seeds) and layer-local reconstruction by two orders of magnitude. The same objective transfers unchanged to 4-bit quantization (1.060 vs. 1.088 for layer-local). Out-of-domain evaluations (C4, CNN/DailyMail) show the advantage of error compensation grows off-domain, while statistics matching keeps feature-statistics discrepancy low off-domain (0.42-0.88 vs. 1.41-2.99 without it), revealing a complementary division of labor between the two mechanisms.