# Nuclear Norm Regularization for Deep Learning

arXiv: https://arxiv.org/abs/2405.14544 · HTML: https://mlsanity.com/paper/2405.14544/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Noise2Noise: Learning Image Restoration without Clean Data](https://mlsanity.com/paper/1803.04189/) (2018)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [Robust Learning with Jacobian Regularization](https://mlsanity.com/paper/1908.02729/) (2019)
- [Robust Large Margin Deep Neural Networks](https://mlsanity.com/paper/1605.08254/) (2016)
- [Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization](https://mlsanity.com/paper/1803.08680/) (2018)
- [Learning Differential Equations that are Easy to Solve](https://mlsanity.com/paper/2007.04504/) (2020)
- [Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --](https://mlsanity.com/paper/2301.03362/) (2023)
- [Gradient Regularization Improves Accuracy of Discriminative Models](https://mlsanity.com/paper/1712.09936/) (2017)
- [OptShrink: An algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkage](https://mlsanity.com/paper/1306.6042/) (2013)

## Built on by (most cited first, 11 of 11)

- [On the Local Complexity of Linear Regions in Deep ReLU Networks](https://mlsanity.com/paper/2412.18283/) (2024)
- [BAGEL: Adversarially Constrained Online Convex Optimization under Separation Oracle Access](https://mlsanity.com/paper/2502.16744/) (2025)
- [Low-Rank Tensor Recovery via Variational Schatten-p Quasi-Norm and Jacobian Regularization](https://mlsanity.com/paper/2506.22134/) (2025)
- [Simplifying Optimal Transport through Schatten-p Regularization](https://mlsanity.com/paper/2510.11910/) (2025)
- [Low-rank MMSE filters, Kronecker-product representation, and regularization: a new perspective](https://mlsanity.com/paper/2512.14932/) (2025)
- [JEDI: Jointly Embedded Inference of Neural Dynamics](https://mlsanity.com/paper/2603.10489/) (2026)
- [Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints](https://mlsanity.com/paper/2603.22824/) (2026)
- [The Geometric Structure of Models Learning Sparse Data](https://mlsanity.com/paper/2605.08464/) (2026)
- [Training-Free Generative Sampling via Moment-Matched Score Smoothing](https://mlsanity.com/paper/2605.14276/) (2026)
- [Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks](https://mlsanity.com/paper/2608.06597/) (2026)
- [On the spectral properties of generative denoiser Jacobians](https://mlsanity.com/paper/2609.36210/) (2026)

## Abstract (arXiv)

Penalizing the nuclear norm of a function's Jacobian encourages it to locally behave like a low-rank linear map. Such functions vary locally along only a handful of directions, making the Jacobian nuclear norm a natural regularizer for machine learning problems. However, this regularizer is intractable for high-dimensional problems, as it requires computing a large Jacobian matrix and taking its singular value decomposition. We show how to efficiently penalize the Jacobian nuclear norm using techniques tailor-made for deep learning. We prove that for functions parametrized as compositions f = g circ h, one may equivalently penalize the average squared Frobenius norm of Jg and Jh. We then propose a denoising-style approximation that avoids the Jacobian computations altogether. Our method is simple, efficient, and accurate, enabling Jacobian nuclear norm regularization to scale to high-dimensional deep learning problems. We complement our theory with an empirical study of our regularizer's performance and investigate applications to denoising and representation learning.