mlsanityData to 7 Oct 2026

Hessian Spectral Analysis at Foundation Model Scale

arXiv 2602.00816 ↗First posted 31 Jan 2026stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful spectral analysis of the true Hessian is tractable at frontier scale. Using shard-local finite-difference Hessian vector products compatible with Fully Sharded Data Parallelism, we perform stochastic Lanczos quadrature on open-source language models with up to 100B parameters, producing the first large-scale spectral density estimates beyond the sub-10B regime. We characterize the numerical behavior of this pipeline, including finite-difference bias, floating-point noise amplification, and their effect on Krylov stability in fp32 and bf16, and derive practical operating regimes that are validated empirically. We further provide end-to-end runtime and memory scaling laws, showing that full-operator spectral probing incurs only a modest constant-factor overhead over first-order training. Crucially, direct access to the Hessian reveals that widely used block-diagonal curvature approximations can fail catastrophically, exhibiting order-one relative error and poor directional alignment even in mid-scale LLMs. Together, our results demonstrate that foundation-model Hessian spectra are both computable and qualitatively misrepresented by prevailing approximations, opening the door to principled curvature-based analysis at scale.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

20 of 40 references.

References · 20

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  11. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  12. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
8 more
  1. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  2. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  3. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  4. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  5. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  6. 2021Analysis of stochastic Lanczos quadrature for spectrum approximationcited by 17
  7. 2025HessFormer: Hessians at Foundation Scalecited by 4
  8. 2025Rethinking LLM Training through Information Geometry and Quantum Metricscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Defense of the Quadratic Modelcited by 6