mlsanityData to 7 Oct 2026

Convexity in ReLU Neural Networks: beyond ICNNs?

arXiv 2501.03017 ↗First posted 6 Jan 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Convex functions and their gradients play a critical role in mathematical imaging, from proximal optimization to Optimal Transport. The successes of deep learning has led many to use learning-based methods, where fixed functions or operators are replaced by learned neural networks. Regardless of their empirical superiority, establishing rigorous guarantees for these methods often requires to impose structural constraints on neural architectures, in particular convexity. The most popular way to do so is to use so-called Input Convex Neural Networks (ICNNs). In order to explore the expressivity of ICNNs, we provide necessary and sufficient conditions for a ReLU neural network to be convex. Such characterizations are based on product of weights and activations, and write nicely for any architecture in the path-lifting framework. As particular applications, we study our characterizations in depth for 1 and 2-hidden-layer neural networks: we show that every convex function implemented by a 1-hidden-layer ReLU network can be also expressed by an ICNN with the same architecture; however this property no longer holds with more layers. Finally, we provide a numerical procedure that allows an exact check of convexity for ReLU neural networks with a large number of affine regions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

31 of 49 references.

References · 31

  1. 2021Convex optimizationcited by 2,479
  2. 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,352
  3. 2020Plug-and-Play Image Restoration with Deep Denoiser Priorcited by 467
  4. 2016Input Convex Neural Networkscited by 407
  5. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  6. 2019Optimal transport mapping via input convex neural networkscited by 202
  7. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  8. 2022Neural Optimal Transportcited by 123
  9. 2017Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problemscited by 107
  10. 2019Wasserstein-2 Generative Networkscited by 96
  11. 2021Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmarkcited by 95
  12. 2018Optimal Control Via Neural Networks: A Convex Approachcited by 82
19 more
  1. 2019Reverse-Engineering Deep ReLU Networkscited by 81
  2. 2020Learning Maximally Monotone Operators for Image Recoverycited by 62
  3. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  4. 2020On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networkscited by 34
  5. 2018A characterization of proximity operatorscited by 32
  6. 2023SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundariescited by 31
  7. 2022On amortizing convex conjugates for optimal transportcited by 30
  8. 2023What's in a Prior? Learned Proximal Networks for Inverse Problemscited by 24
  9. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  10. 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 21
  11. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  12. 2019On approximating nabla f with neural networkscited by 18
  13. 2023Polyhedral Complex Extraction from ReLU Networks using Edge Subdivisioncited by 14
  14. 2024On Minimal Depth in Neural Networkscited by 11
  15. 2023A path-norm toolkit for modern networks: consequences, promises and challengescited by 10
  16. 2022Approximation speed of quantized vs. unquantized ReLU neural networks and beyondcited by 7
  17. 2023Learning Gradients of Convex Functions with Monotone Gradient Networkscited by 7
  18. 2024On a Neural Implementation of Brenier's Polar Factorizationcited by 3
  19. 2014On the convexity of piecewise-defined functionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025On the Depth of Monotone ReLU Neural Networks and ICNNscited by 11
  2. 2025Arithmetic Circuits and Neural Networks for Regular Matroidscited by 1
  3. 2026SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functionscited by 1
  4. 2026Tropical Circuits with Scalar Multiplication Gatescited by 0
  5. 2026Input convex neural networks as surrogates in mathematical optimisationcited by 0
  6. 2026Plug-and-Play Methods Provably Converge Even with Improperly Trained Denoisers: Convergence by Architectural Designcited by 0