# Convexity in ReLU Neural Networks: beyond ICNNs?

arXiv: https://arxiv.org/abs/2501.03017 · HTML: https://mlsanity.com/paper/2501.03017/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 6 Jan 2025.

## Builds on (most cited first, 30 of 31)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Optimal transport mapping via input convex neural networks](https://mlsanity.com/paper/1908.10962/) (2019)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [Neural Optimal Transport](https://mlsanity.com/paper/2201.12220/) (2022)
- [Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems](https://mlsanity.com/paper/1704.03488/) (2017)
- [Wasserstein-2 Generative Networks](https://mlsanity.com/paper/1909.13082/) (2019)
- [Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmark](https://mlsanity.com/paper/2106.01954/) (2021)
- [Optimal Control Via Neural Networks: A Convex Approach](https://mlsanity.com/paper/1805.11835/) (2018)
- [Reverse-Engineering Deep ReLU Networks](https://mlsanity.com/paper/1910.00744/) (2019)
- [Learning Maximally Monotone Operators for Image Recovery](https://mlsanity.com/paper/2012.13247/) (2020)
- [Towards Lower Bounds on the Depth of ReLU Neural Networks](https://mlsanity.com/paper/2105.14835/) (2021)
- [On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks](https://mlsanity.com/paper/2008.09052/) (2020)
- [A characterization of proximity operators](https://mlsanity.com/paper/1807.04014/) (2018)
- [On amortizing convex conjugates for optimal transport](https://mlsanity.com/paper/2210.12153/) (2022)
- [SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries](https://mlsanity.com/paper/2302.12828/) (2023)
- [What's in a Prior? Learned Proximal Networks for Inverse Problems](https://mlsanity.com/paper/2310.14344/) (2023)
- [Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networks](https://mlsanity.com/paper/2207.07696/) (2022)
- [On the Number of Regions of Piecewise Linear Neural Networks](https://mlsanity.com/paper/2206.08615/) (2022)
- [An Embedding of ReLU Networks and an Analysis of their Identifiability](https://mlsanity.com/paper/2107.09370/) (2021)
- [On approximating nabla f with neural networks](https://mlsanity.com/paper/1910.12744/) (2019)
- [Polyhedral Complex Extraction from ReLU Networks using Edge Subdivision](https://mlsanity.com/paper/2306.07212/) (2023)
- [On Minimal Depth in Neural Networks](https://mlsanity.com/paper/2402.15315/) (2024)
- [A path-norm toolkit for modern networks: consequences, promises and challenges](https://mlsanity.com/paper/2310.01225/) (2023)
- [Approximation speed of quantized vs. unquantized ReLU neural networks and beyond](https://mlsanity.com/paper/2205.11874/) (2022)
- [Learning Gradients of Convex Functions with Monotone Gradient Networks](https://mlsanity.com/paper/2301.10862/) (2023)
- [On a Neural Implementation of Brenier's Polar Factorization](https://mlsanity.com/paper/2403.03071/) (2024)

## Built on by (most cited first, 6 of 6)

- [On the Depth of Monotone ReLU Neural Networks and ICNNs](https://mlsanity.com/paper/2505.06169/) (2025)
- [Arithmetic Circuits and Neural Networks for Regular Matroids](https://mlsanity.com/paper/2511.02406/) (2025)
- [SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functions](https://mlsanity.com/paper/2604.22355/) (2026)
- [Tropical Circuits with Scalar Multiplication Gates](https://mlsanity.com/paper/2607.11540/) (2026)
- [Input convex neural networks as surrogates in mathematical optimisation](https://mlsanity.com/paper/2608.09707/) (2026)
- [Plug-and-Play Methods Provably Converge Even with Improperly Trained Denoisers: Convergence by Architectural Design](https://mlsanity.com/paper/2609.33087/) (2026)

## Abstract (arXiv)

Convex functions and their gradients play a critical role in mathematical imaging, from proximal optimization to Optimal Transport. The successes of deep learning has led many to use learning-based methods, where fixed functions or operators are replaced by learned neural networks. Regardless of their empirical superiority, establishing rigorous guarantees for these methods often requires to impose structural constraints on neural architectures, in particular convexity. The most popular way to do so is to use so-called Input Convex Neural Networks (ICNNs). In order to explore the expressivity of ICNNs, we provide necessary and sufficient conditions for a ReLU neural network to be convex. Such characterizations are based on product of weights and activations, and write nicely for any architecture in the path-lifting framework. As particular applications, we study our characterizations in depth for 1 and 2-hidden-layer neural networks: we show that every convex function implemented by a 1-hidden-layer ReLU network can be also expressed by an ICNN with the same architecture; however this property no longer holds with more layers. Finally, we provide a numerical procedure that allows an exact check of convexity for ReLU neural networks with a large number of affine regions.