mlsanityData to 7 Oct 2026

A case for new neural network smoothness constraints

arXiv 2012.07969 ↗First posted 14 Dec 2020stat.ML cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

How sensitive should machine learning models be to input changes? We tackle the question of model smoothness and show that it is a useful inductive bias which aids generalization, adversarial robustness, generative modeling and reinforcement learning. We explore current methods of imposing smoothness constraints and observe they lack the flexibility to adapt to new tasks, they don't account for data modalities, they interact with losses, architectures and optimization in ways not yet fully understood. We conclude that new advances in the field are hinging on finding ways to incorporate data, tasks and learning into our definitions of smoothness.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 8; 2024: 6; 2025: 9; 2026: 2.

43 papers.

Sort:

By area:

  1. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  2. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  3. 2022Learning Smooth Neural Functions via Lipschitz Regularizationcited by 86
  4. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  5. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  6. 2023Towards Stability of Autoregressive Neural Operatorscited by 54
  7. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 49
  8. 2023How Smooth Is Attention?cited by 44
  9. 2021RoMA: Robust Model Adaptation for Offline Model-based Optimizationcited by 39
  10. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  11. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  12. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
31 more
  1. 2023Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothnesscited by 15
  2. 2022Lipschitz Continuity Retained Binary Neural Networkcited by 14
  3. 2024Smoothness Really Matters: A Simple Yet Effective Approach for Unsupervised Graph Domain Adaptationcited by 14
  4. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  5. 2024An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Recordscited by 12
  6. 2021Learning to Assimilate in Chaotic Dynamical Systemscited by 10
  7. 2022Deep Double Descent via Smooth Interpolationcited by 8
  8. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  9. 2021Can convolutional ResNets approximately preserve input distances? A frequency analysis perspectivecited by 7
  10. 2021The Geometric Occam's Razor Implicit in Deep Learningcited by 7
  11. 2022Analyzing Explainer Robustness via Probabilistic Lipschitzness of Prediction Functionscited by 6
  12. 2022Boundary-Aware Uncertainty for Feature Attribution Explainerscited by 6
  13. 2022Learning Globally Smooth Functions on Manifoldscited by 4
  14. 2024The Impact of Geometric Complexity on Neural Collapse in Transfer Learningcited by 4
  15. 2023PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusioncited by 3
  16. 2023Certified Robust Models with Slack Control and Large Lipschitz Constantscited by 3
  17. 2023Towards Architecture-Agnostic Untrained Network Priors for Image Reconstruction with Frequency Regularizationcited by 3
  18. 2025Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regressioncited by 3
  19. 2026Spectral methods: crucial for machine learning, natural for quantum computers?cited by 3
  20. 2023Data Interpolants -- That's What Discriminators in Higher-order Gradient-regularized GANs Arecited by 2
  21. 2025On the Complexity-Faithfulness Trade-off of Gradient-Based Explanationscited by 2
  22. 2022Distributional Gaussian Processes Layers for Out-of-Distribution Detectioncited by 1
  23. 2022Monitoring Vegetation From Space at Extremely Fine Resolutions via Coarsely-Supervised Smooth U-Netcited by 1
  24. 2024From Robustness to Improved Generalization and Calibration in Pre-trained Language Modelscited by 1
  25. 2024Probing Human Visual Robustness with Neurally-Guided Deep Neural Networkscited by 1
  26. 2025RamPINN: Recovering Raman Spectra From Coherent Anti-Stokes Spectra Using Embedded Physicscited by 1
  27. 2025Understanding and Improving Hyperbolic Deep Reinforcement Learningcited by 1
  28. 2022SmOOD: Smoothness-based Out-of-Distribution Detection Approach for Surrogate Neural Networks in Aircraft Designcited by 0
  29. 2025Is data-efficient learning feasible with quantum models?cited by 0
  30. 2025Formal Reasoning About Confidence and Automated Verification of Neural Networkscited by 0
  31. 2026Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamicscited by 0