mlsanityData to 14 Oct 2026

WaterKron and FlipFlop Hessian: Information-Theoretically Grounded Quantization with Kronecker-factored Hessians

arXiv 2609.14706 ↗First posted 13 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

How should a Kronecker-factored Hessian approximation be chosen for post-training quantization? We address this question through WaterKron, which combines two-sided GPTQ with row- and column-dependent waterfilling scales and entropy coding. We derive its high-rate distortion with respect to the full Hessian using an explicit Kronecker-Hessian mismatch factor Phi. This factor quantifies the asymptotic distortion penalty due to the Kronecker Hessian approximation and provides a criterion for selecting the factors optimally. Minimizing Phi leads to a Gaussian covariance-fitting problem with classical ``flip-flop'' updates. We thus give a rate-distortion justification for using the resulting FlipFlop Hessian in quantization. We evaluate it empirically, finding that the FlipFlop Hessian consistently improves KL divergence and perplexity over input-only, marginal, and Frobenius-based Hessian choices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

22 of 27 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  7. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  8. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  9. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  10. 2025Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantizationcited by 28
  11. 2025NestQuant: Nested Lattice Quantization for Matrix Products and LLMscited by 23
  12. 2025Model-Preserving Adaptive Roundingcited by 23
10 more
  1. 2025The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithmcited by 20
  2. 2025Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronoscited by 16
  3. 2024BoA: Attention-aware Post-training Quantization without Backpropagationcited by 12
  4. 2025The Lattice Geometry of Neural Network Quantization -- A Short Equivalence Proof of GPTQ and Babai's Algorithmcited by 9
  5. 2026High-Rate Quantized Matrix Multiplication Icited by 7
  6. 2026WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantizationcited by 7
  7. 2022Core Shrinkage Covariance Estimation for Matrix-variate Datacited by 2
  8. 2026GPTQ-2D: Cubic-Time Two-Sided Adaptive Roundingcited by 2
  9. 2026SoftWater: Class-Aware Rate Allocation for Softmax Quantizationcited by 2
  10. 2026FastKron: Efficient Quantization with Kronecker-Factored Hessianscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KronQ: LLM Quantization via Kronecker-Factored Hessian
  2. 2026HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning
  3. 2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation
  4. 2026Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression
  5. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
  6. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
  7. 2023Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantization
  8. 2026WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points