# Generalizing and Improving Jacobian and Hessian Regularization

arXiv: https://arxiv.org/abs/2212.00311 · HTML: https://mlsanity.com/paper/2212.00311/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Towards Deep Neural Network Architectures Robust to Adversarial Examples](https://mlsanity.com/paper/1412.5068/) (2014)
- [Robust Learning with Jacobian Regularization](https://mlsanity.com/paper/1908.02729/) (2019)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)
- [Gradient Regularization Improves Accuracy of Discriminative Models](https://mlsanity.com/paper/1712.09936/) (2017)
- [Neural Networks with Cheap Differential Operators](https://mlsanity.com/paper/1912.03579/) (2019)
- [Input Hessian Regularization of Neural Networks](https://mlsanity.com/paper/2009.06571/) (2020)
- [Exact Spectral Norm Regularization for Neural Networks](https://mlsanity.com/paper/2206.13581/) (2022)

## Built on by (most cited first, 2 of 2)

- [On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models](https://mlsanity.com/paper/2402.03845/) (2024)
- [Smoothness-Based Derandomization of PAC-Bayes Bounds](https://mlsanity.com/paper/2606.19105/) (2026)

## Abstract (arXiv)

Jacobian and Hessian regularization aim to reduce the magnitude of the first and second-order partial derivatives with respect to neural network inputs, and they are predominantly used to ensure the adversarial robustness of image classifiers. In this work, we generalize previous efforts by extending the target matrix from zero to any matrix that admits efficient matrix-vector products. The proposed paradigm allows us to construct novel regularization terms that enforce symmetry or diagonality on square Jacobian and Hessian matrices. On the other hand, the major challenge for Jacobian and Hessian regularization has been high computational complexity. We introduce Lanczos-based spectral norm minimization to tackle this difficulty. This technique uses a parallelized implementation of the Lanczos algorithm and is capable of effective and stable regularization of large Jacobian and Hessian matrices. Theoretical justifications and empirical evidence are provided for the proposed paradigm and technique. We carry out exploratory experiments to validate the effectiveness of our novel regularization terms. We also conduct comparative experiments to evaluate Lanczos-based spectral norm minimization against prior methods. Results show that the proposed methodologies are advantageous for a wide range of tasks.