mlsanityData to 7 Oct 2026

Dissecting Quantization Error: A Concentration-Alignment Perspective

arXiv 2603.04359 ↗First posted 4 Mar 2026cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 4 Mar 2026.

Builds on

18 of 26 references.

References · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  11. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
6 more
  1. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  2. 2024Massive Activations in Large Language Modelscited by 277
  3. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  4. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 94
  5. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  6. 2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Contraction-Gauge Preconditioning for Quantized Matrix Multiplicationcited by 1
  2. 2026Quantization Degradation in Large Language Models: A Signal-Noise Perspectivecited by 0
  3. 2026KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Models
  2. 2026Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models
  3. 2026Rethinking Practical and Efficient Quantization Calibration for Vision-Language Models
  4. 2026Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
  5. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospects
  6. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
  7. 2026AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation
  8. 2026Stable FP4 Training via Transposition-Invariant Block Quantization