# The Curious Case of Convex Neural Networks

arXiv: https://arxiv.org/abs/2006.05103 · HTML: https://mlsanity.com/paper/2006.05103/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 16 of 16)

- [Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions](https://mlsanity.com/paper/2202.01331/) (2022)
- [Principled Weight Initialisation for Input-Convex Neural Networks](https://mlsanity.com/paper/2312.12474/) (2023)
- [Asymmetric Certified Robustness via Feature-Convex Neural Networks](https://mlsanity.com/paper/2302.01961/) (2023)
- [CDiNN -Convex Difference Neural Networks](https://mlsanity.com/paper/2103.17231/) (2021)
- [Reappraising Domain Generalization in Neural Networks](https://mlsanity.com/paper/2110.07981/) (2021)
- [PolyMPCNet: Towards ReLU-free Neural Architecture Search in Two-party Computation Based Private Inference](https://mlsanity.com/paper/2209.09424/) (2022)
- [Defining Neural Network Architecture through Polytope Structures of Dataset](https://mlsanity.com/paper/2402.02407/) (2024)
- [SPIEDiff: robust learning of long-time macroscopic dynamics from short-time particle simulations with quantified epistemic uncertainty](https://mlsanity.com/paper/2505.13501/) (2025)
- [A mathematical certification for positivity conditions in Neural Networks with applications to partial monotonicity and Trustworthy AI](https://mlsanity.com/paper/2406.08525/) (2024)
- [Knowledge-based Neural Ordinary Differential Equations for Cosserat Rod-based Soft Robots](https://mlsanity.com/paper/2408.07776/) (2024)
- [Gradient Networks](https://mlsanity.com/paper/2404.07361/) (2024)
- [Convex Is Back: Solving Belief MDPs With Convexity-Informed Deep Reinforcement Learning](https://mlsanity.com/paper/2502.09298/) (2025)
- [Certifiably robust malware detectors by design](https://mlsanity.com/paper/2508.10038/) (2025)
- [Hidden Monotonicity: Explaining Deep Neural Networks via their DC Decomposition](https://mlsanity.com/paper/2601.07700/) (2026)
- [Neural Slack Variables for Shape Constraints](https://mlsanity.com/paper/2606.13803/) (2026)
- [Brenier Meets Adversarial Training: Optimal Transport Geometry for Robust Learning](https://mlsanity.com/paper/2609.31363/) (2026)

## Abstract (arXiv)

In this paper, we investigate a constrained formulation of neural networks where the output is a convex function of the input. We show that the convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The convexity constraints include restricting the weights (for all but the first layer) to be non-negative and using a non-decreasing convex activation function. Albeit simple, these constraints have profound implications on the generalization abilities of the network. We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi layer perceptrons and achieve similar performance as compared to base convolutional architectures and (c) IOC-NNs show robustness to noise in train labels. We demonstrate the efficacy of the proposed idea using thorough experiments and ablation studies on standard image classification datasets with three different neural network architectures.