# Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations

arXiv: https://arxiv.org/abs/2505.02537 · HTML: https://mlsanity.com/paper/2505.02537/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 5 May 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Fully Neural Network based Model for General Temporal Point Processes](https://mlsanity.com/paper/1905.09690/) (2019)
- [Unconstrained Monotonic Neural Networks](https://mlsanity.com/paper/1908.05164/) (2019)
- [Continuously Differentiable Exponential Linear Units](https://mlsanity.com/paper/1704.07483/) (2017)
- [Deep Lattice Networks and Partial Monotonic Functions](https://mlsanity.com/paper/1709.06680/) (2017)
- [Monotonic Calibrated Interpolated Look-Up Tables](https://mlsanity.com/paper/1505.06378/) (2015)
- [Counterexample-Guided Learning of Monotonic Neural Networks](https://mlsanity.com/paper/2006.08852/) (2020)
- [Deep Learning with S-shaped Rectified Linear Activation Units](https://mlsanity.com/paper/1512.07030/) (2015)
- [Neural Likelihoods via Cumulative Distribution Functions](https://mlsanity.com/paper/1811.00974/) (2018)
- [How to Incorporate Monotonicity in Deep Networks While Preserving Flexibility?](https://mlsanity.com/paper/1909.10662/) (2019)
- [Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks](https://mlsanity.com/paper/2010.09458/) (2020)
- [Neural Fine-Gray: Monotonic neural networks for competing risks](https://mlsanity.com/paper/2305.06703/) (2023)
- [Size and depth of monotone neural networks: interpolation and approximation](https://mlsanity.com/paper/2207.05275/) (2022)
- [Expressive Monotonic Neural Networks](https://mlsanity.com/paper/2307.07512/) (2023)
- [MonoNet: Towards Interpretable Models by Learning Monotonic Features](https://mlsanity.com/paper/1909.13611/) (2019)

## Built on by (most cited first, 8 of 8)

- [AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery](https://mlsanity.com/paper/2604.05550/) (2026)
- [Embedding Foundation Model Predictions in Discrete-Choice Models with Structural Guarantees](https://mlsanity.com/paper/2606.26432/) (2026)
- [Monotonic Kolmogorov-Arnold Networks: A Theoretical and Empirical Study of Monotonicity as an Inductive Bias](https://mlsanity.com/paper/2606.17886/) (2026)
- [Monotonicity as an Architectural Bias for Robust Language Models](https://mlsanity.com/paper/2602.02686/) (2026)
- [Functional Similarity Metric for Neural Networks: Overcoming Parametric Ambiguity via Activation Region Analysis](https://mlsanity.com/paper/2604.16426/) (2026)
- [Neural Slack Variables for Shape Constraints](https://mlsanity.com/paper/2606.13803/) (2026)
- [LRX-PINN: A Layer-Resolving XNet Physics-Informed Neural Network with Integrated Cauchy Activations for Convection-Dominated Problems](https://mlsanity.com/paper/2607.03682/) (2026)
- [Differentiable Fuzzy Inference Layer: A Monotone, Compositional Ordinal Reasoning Head for Large Language Models](https://mlsanity.com/paper/2609.26113/) (2026)

## Abstract (arXiv)

Conventional techniques for imposing monotonicity in MLPs by construction involve the use of non-negative weight constraints and bounded activation functions, which pose well-known optimization challenges. In this work, we generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activations that saturate on alternating sides are universal approximators for monotonic functions. Additionally, we show an equivalence between the saturation side in the activations and the sign of the weight constraint. This connection allows us to prove that MLPs with convex monotone activations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts. Our results provide theoretical grounding to the empirical effectiveness observed in previous works while leading to possible architectural simplification. Moreover, to further alleviate the optimization difficulties, we propose an alternative formulation that allows the network to adjust its activations according to the sign of the weights. This eliminates the requirement for weight reparameterization, easing initialization and improving training stability. Experimental evaluation reinforces the validity of the theoretical results, showing that our novel approach compares favourably to traditional monotonic architectures.