# Gradient Regularization Improves Accuracy of Discriminative Models

arXiv: https://arxiv.org/abs/1712.09936 · HTML: https://mlsanity.com/paper/1712.09936/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2017.

## Built on by (most cited first, 26 of 26)

- [Learning Smooth Neural Functions via Lipschitz Regularization](https://mlsanity.com/paper/2202.08345/) (2022)
- [On Linear Stability of SGD and Input-Smoothness of Neural Networks](https://mlsanity.com/paper/2105.13462/) (2021)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability](https://mlsanity.com/paper/2307.07873/) (2023)
- [Local Disentanglement in Variational Auto-Encoders Using Jacobian L1 Regularization](https://mlsanity.com/paper/2106.02923/) (2021)
- [PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization](https://mlsanity.com/paper/2409.17137/) (2024)
- [Why Adversarial Reprogramming Works, When It Fails, and How to Tell the Difference](https://mlsanity.com/paper/2108.11673/) (2021)
- [Nuclear Norm Regularization for Deep Learning](https://mlsanity.com/paper/2405.14544/) (2024)
- [PAC-Bayes-Chernoff bounds for unbounded losses](https://mlsanity.com/paper/2401.01148/) (2024)
- [Nash Equilibria, Regularization and Computation in Optimal Transport-Based Distributionally Robust Optimization](https://mlsanity.com/paper/2303.03900/) (2023)
- [The Geometric Occam's Razor Implicit in Deep Learning](https://mlsanity.com/paper/2111.15090/) (2021)
- [Jacobian Regularization for Mitigating Universal Adversarial Perturbations](https://mlsanity.com/paper/2104.10459/) (2021)
- [MIRACLE3D: Memory-efficient Integrated Robust Approach for Continual Learning on Point Clouds via Shape Model Construction](https://mlsanity.com/paper/2410.06418/) (2024)
- [Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization](https://mlsanity.com/paper/2204.02485/) (2022)
- [Machines of finite depth: towards a formalization of neural networks](https://mlsanity.com/paper/2204.12786/) (2022)
- [Understanding Open-Set Recognition by Jacobian Norm and Inter-Class Separation](https://mlsanity.com/paper/2209.11436/) (2022)
- [Generalizing and Improving Jacobian and Hessian Regularization](https://mlsanity.com/paper/2212.00311/) (2022)
- [From Robustness to Improved Generalization and Calibration in Pre-trained Language Models](https://mlsanity.com/paper/2404.00758/) (2024)
- [Universal Robustness via Median Randomized Smoothing for Real-World Super-Resolution](https://mlsanity.com/paper/2405.14934/) (2024)
- [Adversarially robust generalization theory via Jacobian regularization for deep neural networks](https://mlsanity.com/paper/2412.12449/) (2024)
- [Smoothness-Based Derandomization of PAC-Bayes Bounds](https://mlsanity.com/paper/2606.19105/) (2026)
- [Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy](https://mlsanity.com/paper/2302.04369/) (2023)
- [On Continuity of Robust and Accurate Classifiers](https://mlsanity.com/paper/2309.17048/) (2023)
- [Exploring Biologically Inspired Mechanisms of Adversarial Robustness](https://mlsanity.com/paper/2405.00679/) (2024)
- [CMR: Contractive Mapping Embeddings for Robust Humanoid Locomotion on Unstructured Terrains](https://mlsanity.com/paper/2602.03511/) (2026)
- [SaliencyDecor: Enhancing Neural Network Interpretability through Feature Decorrelation](https://mlsanity.com/paper/2604.25315/) (2026)

## Abstract (arXiv)

Regularizing the gradient norm of the output of a neural network with respect to its inputs is a powerful technique, rediscovered several times. This paper presents evidence that gradient regularization can consistently improve classification accuracy on vision tasks, using modern deep neural networks, especially when the amount of training data is small. We introduce our regularizers as members of a broader class of Jacobian-based regularizers. We demonstrate empirically on real and synthetic data that the learning process leads to gradients controlled beyond the training points, and results in solutions that generalize well.