mlsanityData to 7 Oct 2026

Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace

arXiv 2208.05924 ↗First posted 11 Aug 2022cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we develop a novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by a recent guarantee bound of the generalization error. We explain its benefits in finding flat minima and avoiding Lyapunov stability in dynamical systems. We adopt the Hutchinson method as a classical unbiased estimator for the trace of a matrix and further accelerate its calculation using a dropout scheme. Experiments demonstrate that our method outperforms existing regularizers and data augmentation methods, such as Jacobian, Confidence Penalty, Label Smoothing, Cutout, and Mixup.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 11 Aug 2022.

Builds on

26 of 40 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2016Wide Residual Networkscited by 3,057
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  12. 2017Sharp Minima Can Generalize For Deep Netscited by 462
14 more
  1. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  2. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  3. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  4. 2018DropBlock: A regularization method for convolutional networkscited by 261
  5. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  6. 2019Robust Learning with Jacobian Regularizationcited by 127
  7. 2016Robust Large Margin Deep Neural Networkscited by 117
  8. 2013Dropout Training as Adaptive Regularizationcited by 89
  9. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  10. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  11. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 43
  12. 2014On the Inductive Bias of Dropoutcited by 19
  13. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
  14. 2019Deep Learning for Detecting Building Defects Using Convolutional Neural Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Smoothness-Based Derandomization of PAC-Bayes Boundscited by 1
  2. 2025Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapescited by 0
  3. 2026X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correctioncited by 0
  4. 2026How Far Can Sharpness and Complexity Jointly Explain Generalization?cited by 0