mlsanityData to 7 Oct 2026

HessFormer: Hessians at Foundation Scale

arXiv 2505.11564 ↗First posted 16 May 2025cs.LG stat.MLcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Whilst there have been major advancements in the field of first order optimisation of deep learning models, where state of the art open source mixture of expert models go into the hundreds of billions of parameters, methods that rely on Hessian vector products, are still limited to run on a single GPU and thus cannot even work for models in the billion parameter range. We release a software package HessFormer, which integrates nicely with the well known Transformers package and allows for distributed hessian vector computation across a single node with multiple GPUs. Underpinning our implementation is a distributed stochastic lanczos quadrature algorithm, which we release for public consumption. Using this package we investigate the Hessian spectral density of the recent Deepseek 70bn parameter model.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 20 of its 28 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

20 of 28 references.

References · 20

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  11. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  12. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
8 more
  1. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  2. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  3. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  4. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  5. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  6. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  7. 2025Phase Transitions in Large Language Models and the O(N) Modelcited by 11
  8. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026A Defense of the Quadratic Modelcited by 6
  2. 2025Mining Useful General Data for Low-Resource Domain Adaptationcited by 3
  3. 2026Hessian Spectral Analysis at Foundation Model Scalecited by 1
  4. 2026Taming Curvature: Architecture Warm-Up for Stable Transformer Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approach
  2. 2025Decentralized Diffusion Models
  3. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative
  4. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
  5. 2026SoftServe: A Scalable Quasi-Newton Method for Deep Learning
  6. 2026To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
  7. 2026Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count
  8. 2025Paris: A Decentralized Trained Open-Weight Diffusion Model