mlsanityData to 14 Oct 2026

Gradient Networks

arXiv 2404.07361 ↗First posted 10 Apr 2024cs.LG cs.NE eess.SPcited by 0

Not on the map. Why?

Abstract (arXiv)

Directly parameterizing and learning gradients of functions has widespread significance, with specific applications in inverse problems, generative modeling, and optimal transport. This paper introduces gradient networks (GradNets): novel neural network architectures that parameterize gradients of various function classes. GradNets exhibit specialized architectural constraints that ensure correspondence to gradient functions. We provide a comprehensive GradNet design framework that includes methods for transforming GradNets into monotone gradient networks (mGradNets), which are guaranteed to represent gradients of convex functions. Our results establish that our proposed GradNet (and mGradNet) universally approximate the gradients of (convex) functions. Furthermore, these networks can be customized to correspond to specific spaces of potential functions, including transformed sums of (convex) ridge functions. Our analysis leads to two distinct GradNet architectures, GradNet-C and GradNet-M, and we describe the corresponding monotone versions, mGradNet-C and mGradNet-M. Our empirical results demonstrate that these architectures provide efficient parameterizations and outperform existing methods by up to 15 dB in gradient field tasks and by up to 11 dB in Hamiltonian dynamics learning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

37 of 63 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2021Convex optimizationcited by 2,515
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  7. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  8. 2019Hamiltonian Neural Networkscited by 712
  9. 2016Learning to learn by gradient descent by gradient descentcited by 620
  10. 2016Input Convex Neural Networkscited by 414
  11. 2016The Little Engine that Could: Regularization by Denoising (RED)cited by 307
  12. 2020Learning Gradient Fields for Shape Generationcited by 245
25 more
  1. 2019Optimal transport mapping via input convex neural networkscited by 206
  2. 2020AutoInt: Automatic Integration for Fast Neural Volume Renderingcited by 204
  3. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 198
  4. 2021Learning Gradient Fields for Molecular Conformation Generationcited by 163
  5. 2017Sobolev Training for Neural Networkscited by 143
  6. 2021Gradient Step Denoiser for convergent Plug-and-Playcited by 108
  7. 2018Regularization by Denoising: Clarifications and New Interpretationscited by 103
  8. 2021Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmarkcited by 96
  9. 2022Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imagingcited by 96
  10. 2020Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimizationcited by 90
  11. 2021Gradients are Not All You Needcited by 86
  12. 2018Optimal Control Via Neural Networks: A Convex Approachcited by 85
  13. 2022Supervised Training of Conditional Monge Mapscited by 61
  14. 2020Learned convex regularizers for inverse problemscited by 55
  15. 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 41
  16. 2023Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithmscited by 31
  17. 2014Insights into analysis operator learning: From patch-based sparse models to higher-order MRFscited by 18
  18. 2019On approximating nabla f with neural networkscited by 18
  19. 2021Input Convex Gradient Networkscited by 17
  20. 2020The Curious Case of Convex Neural Networkscited by 16
  21. 2023Principled Weight Initialisation for Input-Convex Neural Networkscited by 12
  22. 2021On Plug-and-Play Regularization using Linear Denoiserscited by 9
  23. 2022PnP-ReG: Learned Regularizing Gradient for Plug-and-Play Gradient Descentcited by 7
  24. 2023Learning Gradients of Convex Functions with Monotone Gradient Networkscited by 7
  25. 2017Discriminative Optimization: Theory and Applications to Computer Vision Problemscited by 1

Built on by

Not cited yet.