mlsanityData to 14 Oct 2026

Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models

arXiv 2607.14630 ↗First posted 16 Jul 2026cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Layer-wise post-training quantization of large language models minimizes each layer's reconstruction error in isolation, allowing quantization errors to accumulate across depth and causing severe degradation in extreme low-bit regimes. We formulate quantization as a joint optimization over the discrete codes and scales of all layers, driven by two mechanisms: (i) cross-layer error compensation, which maintains the network-level accumulated error through the recursion e_{l+1} = A_l e_l + q_l, with a propagation operator A_l derived from the layer's input differential and a local quantization residual q_l evaluated at teacher features; and (ii) finite-sample feature-statistics matching, which aligns means, projected covariances, and centered empirical kernels between the full-precision and quantized networks under relative normalization. We prove that instantiating the propagation operator as a finite difference of the quantized network makes the recursion exact for arbitrary nonlinear layers, enabling an efficient forward-difference implementation. Binary weights are optimized via a mirror-descent parameterization u = tanh(beta*z) with annealed inverse temperature and group-wise log-scales. On Qwen2.5-1.5B with 1.125-bit group-binary weights, error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher, outperforming logit distillation (14.09 +/- 0.53; 32 percent relative, more than 8 sigma over 3 seeds) and layer-local reconstruction by two orders of magnitude. The same objective transfers unchanged to 4-bit quantization (1.060 vs. 1.088 for layer-local). Out-of-domain evaluations (C4, CNN/DailyMail) show the advantage of error compensation grows off-domain, while statistics matching keeps feature-statistics discrepancy low off-domain (0.42-0.88 vs. 1.41-2.99 without it), revealing a complementary division of labor between the two mechanisms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 24 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

22 of 24 references.

References · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  5. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  6. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  7. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  8. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  12. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
10 more
  1. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  2. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  3. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  4. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 59
  5. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  6. 2025Model-Preserving Adaptive Roundingcited by 23
  7. 2019Mirror Descent View for Neural Network Quantizationcited by 16
  8. 2026Rethinking Residual Errors in Compensation-based LLM Quantizationcited by 7
  9. 2023Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Datacited by 5
  10. 2026TurboBoA: Faster and Exact Attention-aware Quantization without Backpropagationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels
  2. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
  3. 2025Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
  4. 2025Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
  5. 2024LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment
  6. 2024ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization
  7. 2023QuantEase: Optimization-based Quantization for Language Models
  8. 2026RDQ: Residual Distribution Quantization for Large Language Models