# A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizations

arXiv: https://arxiv.org/abs/2407.06888 · HTML: https://mlsanity.com/paper/2407.06888/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 77 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Efficient Neural Network Robustness Certification with General Activation Functions](https://mlsanity.com/paper/1811.00866/) (2018)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming](https://mlsanity.com/paper/1903.01287/) (2019)
- [A Unified Algebraic Perspective on Lipschitz Neural Networks](https://mlsanity.com/paper/2303.03169/) (2023)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [Lipschitz Bounded Equilibrium Networks](https://mlsanity.com/paper/2010.01732/) (2020)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [Stability Analysis using Quadratic Constraints for Systems with Neural Network Controllers](https://mlsanity.com/paper/2006.07579/) (2020)
- [Approximating Lipschitz continuous functions with GroupSort neural networks](https://mlsanity.com/paper/2006.05254/) (2020)
- [Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularization](https://mlsanity.com/paper/2310.00116/) (2023)
- [A Quantitative Geometric Approach to Neural-Network Smoothness](https://mlsanity.com/paper/2203.01212/) (2022)
- [Lipschitz constant estimation for 1D convolutional neural networks](https://mlsanity.com/paper/2211.15253/) (2022)
- [Sparse Polynomial Optimisation for Neural Network Verification](https://mlsanity.com/paper/2202.02241/) (2022)
- [Robust Machine Learning Modeling for Predictive Control Using Lipschitz-Constrained Neural Networks](https://mlsanity.com/paper/2308.13721/) (2023)
- [Lipschitz constant estimation for general neural network architectures using control tools](https://mlsanity.com/paper/2405.01125/) (2024)
- [Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations](https://mlsanity.com/paper/2401.14033/) (2024)
- [Dissipativity and Integral Quadratic Constraints, Tailored computational robustness tests for complex interconnections](https://mlsanity.com/paper/2105.07401/) (2021)
- [Convex searches for discrete-time Zames-Falb multipliers](https://mlsanity.com/paper/1812.02397/) (2018)
- [Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networks](https://mlsanity.com/paper/2405.05236/) (2024)
- [Reduced-Order Neural Network Synthesis with Robustness Guarantees](https://mlsanity.com/paper/2102.09284/) (2021)
- [Stability Analysis of Recurrent Neural Networks by IQC with Copositive Mutipliers](https://mlsanity.com/paper/2202.04592/) (2022)

## Built on by (most cited first, 2 of 2)

- [Local Stability and Region of Attraction Analysis for Neural Network Feedback Systems under Positivity Constraints](https://mlsanity.com/paper/2505.22889/) (2025)
- [Quadratic Characterizations for Reachability Analysis of Neural Networks](https://mlsanity.com/paper/2605.20482/) (2026)

## Abstract (arXiv)

This paper derives a complete set of quadratic constraints (QCs) for the repeated ReLU. The complete set of QCs is described by a collection of matrix copositivity conditions. We also show that only two functions satisfy all QCs in our complete set: the repeated ReLU and flipped ReLU. Thus our complete set of QCs bounds the repeated ReLU as tight as possible up to the sign invariance inherent in quadratic forms. We derive a similar complete set of incremental QCs for repeated ReLU, which can potentially lead to less conservative Lipschitz bounds for ReLU networks than the standard LipSDP approach. The basic constructions are also used to derive the complete sets of QCs for other piecewise linear activation functions such as leaky ReLU, MaxMin, and HouseHolder. Finally, we illustrate the use of the complete set of QCs to assess stability and performance for recurrent neural networks with ReLU activation functions. We rely on a standard copositivity relaxation to formulate the stability/performance condition as a semidefinite program. Simple examples are provided to illustrate that the complete sets of QCs and incremental QCs can yield less conservative bounds than existing sets.