mlsanityData to 14 Oct 2026

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

arXiv 2609.02451 ↗First posted 2 Sep 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a scalable Kronecker-based approximation that captures cross-layer interactions without storing the entire Fisher matrix, enabling practical Hessian analysis for billion-parameter networks where full computation is infeasible. Our approach reveals consistent vulnerability patterns: value projection layers exhibit the highest sensitivity and strongest cross-layer correlations across multiple model families, while other components exhibit architecture-specific behaviors. Through extensive experiments on quantization, sparsification, inter-layer corruption, and post-corruption fine-tuning, we demonstrate that our approximation strongly correlates with both performance degradation and recovery. Our framework provides a practical, theoretically grounded tool for identifying fragile components in large models, opening new avenues for guided compression and optimization strategies, such as mixed-precision allocation, layer-wise sparsity, and adaptive low-rank decomposition across layers and even individual weight groups.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

18 of 34 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  8. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  9. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  10. 2022Language model compression with weighted low-rank factorizationcited by 195
  11. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  12. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
6 more
  1. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  2. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  3. 2019Modular Block-diagonal Curvature Approximations for Feedforward Architecturescited by 13
  4. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  5. 2017Block-diagonal Hessian-free Optimization for Training Neural Networkscited by 10
  6. 2017Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Formcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HAS-VQ: Hessian-Adaptive Sparse Vector Quantization for High-Fidelity LLM Compression
  2. 2026KronQ: LLM Quantization via Kronecker-Factored Hessian
  3. 2026Break Through the Compression Bottleneck: From Theory to Practice
  4. 2026Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
  5. 2026On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
  6. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
  7. 2024Data-free Weight Compress and Denoise for Large Language Models
  8. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Size