# Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach

arXiv: https://arxiv.org/abs/2310.16647 · HTML: https://mlsanity.com/paper/2310.16647/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Coordinate Descent Algorithms](https://mlsanity.com/paper/1502.04759/) (2015)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [Predicting AC Optimal Power Flows: Combining Deep Learning and Lagrangian Dual Methods](https://mlsanity.com/paper/1909.10461/) (2019)
- [Lagrangian Duality for Constrained Deep Learning](https://mlsanity.com/paper/2001.09394/) (2020)
- [Enhanced Physics-Informed Neural Networks with Augmented Lagrangian Relaxation Method (AL-PINNs)](https://mlsanity.com/paper/2205.01059/) (2022)
- [Towards a topological-geometrical theory of group equivariant non-expansive operators for data analysis and machine learning](https://mlsanity.com/paper/1812.11832/) (2018)
- [Training neural networks under physical constraints using a stochastic augmented Lagrangian approach](https://mlsanity.com/paper/2009.07330/) (2020)
- [Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes](https://mlsanity.com/paper/2102.12894/) (2021)
- [Progressive Weight Pruning of Deep Neural Networks using ADMM](https://mlsanity.com/paper/1810.07378/) (2018)
- [An ADMM-Based Universal Framework for Adversarial Attacks on Deep Neural Networks](https://mlsanity.com/paper/1804.03193/) (2018)
- [Lagrangian dual framework for conservative neural network solutions of kinetic equations](https://mlsanity.com/paper/2106.12147/) (2021)

## Built on by (most cited first, 4 of 4)

- [Generalized Lagrangian Neural Networks](https://mlsanity.com/paper/2401.03728/) (2024)
- [CausalWrap: Model-Agnostic Causal Constraint Wrappers for Tabular Synthetic Data](https://mlsanity.com/paper/2603.02015/) (2026)
- [Sample-wise Constrained Learning via a Sequential Penalty Approach with Applications in Image Processing](https://mlsanity.com/paper/2601.16812/) (2026)
- [ORACLE: Optimizer-Relative Alignment for Constrained LEarning](https://mlsanity.com/paper/2610.09040/) (2026)

## Abstract (arXiv)

Regularizing Deep Neural Networks (DNNs) is essential for improving generalizability and preventing overfitting. Fixed penalty methods, though common, lack adaptability and suffer from hyperparameter sensitivity. In this paper, we propose a novel approach to DNN regularization by framing the training process as a constrained optimization problem. Where the data fidelity term is the minimization objective and the regularization terms serve as constraints. Then, we employ the Stochastic Augmented Lagrangian (SAL) method to achieve a more flexible and efficient regularization mechanism. Our approach extends beyond black-box regularization, demonstrating significant improvements in white-box models, where weights are often subject to hard constraints to ensure interpretability. Experimental results on image-based classification on MNIST, CIFAR10, and CIFAR100 datasets validate the effectiveness of our approach. SAL consistently achieves higher Accuracy while also achieving better constraint satisfaction, thus showcasing its potential for optimizing DNNs under constrained settings.