Data to 14 Oct 2026

PyHessian: Neural Networks Through the Lens of the Hessian

arXiv 1912.07145 ↗First posted 16 Dec 2019cs.LG cs.NA math.NAcited by 292

Not on the map. Why?

Abstract (arXiv)

We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density, and it supports distributed-memory execution on cloud/supercomputer systems and is available as open source. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. To illustrate this, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape smoother, thus making it easier for Stochastic Gradient Descent to converge to a good solution. Our extensive analysis shows new finer-scale insights, demonstrating that, while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization does not necessarily make the loss landscape smoother, especially for shallower networks.

Key facts. Cited by 292 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 30; 2022: 51; 2023: 51; 2024: 57; 2025: 43; 2026: 60.

292 papers.

Sort:

By area:

2021Characterizing possible failure modes in physics-informed neural networkscited by 643
2023Fine-Tuning Language Models with Just Forward Passescited by 345
2022How Do Vision Transformers Work?cited by 307
2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
2021Deep Learning Through the Lens of Example Difficultycited by 141
2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122

38 more

2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retrainingcited by 118
2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 113
2022Neurotoxin: Durable Backdoors in Federated Learningcited by 98
2021Network Quantization with Element-wise Gradient Scalingcited by 87
2022A Closer Look at Smoothness in Domain Adversarial Trainingcited by 85
2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
2022When Do Flat Minima Optimizers Work?cited by 78
2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
2023PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networkscited by 64
2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
2022Advancing Model Pruning via Bi-level Optimizationcited by 56
2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 56
2022Neuromorphic Data Augmentation for Training Spiking Neural Networkscited by 52
2021Hessian-Aware Pruning and Optimal Neural Implantcited by 45
2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
2022Anticorrelated Noise Injection for Improved Generalizationcited by 42
2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inferencecited by 42
2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
2022LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinitycited by 39
2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
2022On the Soft-Subnetwork for Few-shot Class Incremental Learningcited by 37
2024Make Continual Learning Stronger via C-Flatcited by 35
2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
2023ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognitioncited by 31
2022Escaping Saddle Points for Effective Generalization on Class-Imbalanced Datacited by 28
2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 28
2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 27

50 most cited of 292 All 292 (JSON)