# Principled Weight Initialisation for Input-Convex Neural Networks

arXiv: https://arxiv.org/abs/2312.12474 · HTML: https://mlsanity.com/paper/2312.12474/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Optimal transport mapping via input convex neural networks](https://mlsanity.com/paper/1908.10962/) (2019)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity](https://mlsanity.com/paper/1602.05897/) (2016)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [Mean Field Residual Networks: On the Edge of Chaos](https://mlsanity.com/paper/1712.08969/) (2017)
- [Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping](https://mlsanity.com/paper/2110.01765/) (2021)
- [Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers](https://mlsanity.com/paper/2203.08120/) (2022)
- [Principled Weight Initialization for Hypernetworks](https://mlsanity.com/paper/2312.08399/) (2023)
- [The Curious Case of Convex Neural Networks](https://mlsanity.com/paper/2006.05103/) (2020)
- [CDiNN -Convex Difference Neural Networks](https://mlsanity.com/paper/2103.17231/) (2021)
- [Learning Invariances with Generalised Input-Convex Neural Networks](https://mlsanity.com/paper/2204.07009/) (2022)

## Built on by (most cited first, 12 of 12)

- [Optimal Flow Matching: Learning Straight Trajectories in Just One Step](https://mlsanity.com/paper/2403.13117/) (2024)
- [GNN-VPA: A Variance-Preserving Aggregation Strategy for Graph Neural Networks](https://mlsanity.com/paper/2403.04747/) (2024)
- [Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks](https://mlsanity.com/paper/2505.11881/) (2025)
- [DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation](https://mlsanity.com/paper/2409.14307/) (2024)
- [Amortized Maximum Inner Product Search with Learned Support Functions](https://mlsanity.com/paper/2603.08001/) (2026)
- [SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functions](https://mlsanity.com/paper/2604.22355/) (2026)
- [Monotonic Kolmogorov-Arnold Networks: A Theoretical and Empirical Study of Monotonicity as an Inductive Bias](https://mlsanity.com/paper/2606.17886/) (2026)
- [Gradient Networks](https://mlsanity.com/paper/2404.07361/) (2024)
- [Beyond Gaussian Initializations: Signal Preserving Weight Initialization for Odd-Sigmoid Activations](https://mlsanity.com/paper/2509.23085/) (2025)
- [A lift for input-convex neural net training](https://mlsanity.com/paper/2605.24274/) (2026)
- [Neural Slack Variables for Shape Constraints](https://mlsanity.com/paper/2606.13803/) (2026)
- [Brenier Meets Adversarial Training: Optimal Transport Geometry for Robust Learning](https://mlsanity.com/paper/2609.31363/) (2026)

## Abstract (arXiv)

Input-Convex Neural Networks (ICNNs) are networks that guarantee convexity in their input-output mapping. These networks have been successfully applied for energy-based modelling, optimal transport problems and learning invariances. The convexity of ICNNs is achieved by using non-decreasing convex activation functions and non-negative weights. Because of these peculiarities, previous initialisation strategies, which implicitly assume centred weights, are not effective for ICNNs. By studying signal propagation through layers with non-negative weights, we are able to derive a principled weight initialisation for ICNNs. Concretely, we generalise signal propagation theory by removing the assumption that weights are sampled from a centred distribution. In a set of experiments, we demonstrate that our principled initialisation effectively accelerates learning in ICNNs and leads to better generalisation. Moreover, we find that, in contrast to common belief, ICNNs can be trained without skip-connections when initialised correctly. Finally, we apply ICNNs to a real-world drug discovery task and show that they allow for more effective molecular latent space exploration.