# Stochastic Penalty-Barrier Method for Constrained Machine Learning

arXiv: https://arxiv.org/abs/2605.18618 · HTML: https://mlsanity.com/paper/2605.18618/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 109 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10561/) (2017)
- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks](https://mlsanity.com/paper/2402.09092/) (2024)
- [A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians](https://mlsanity.com/paper/2106.13015/) (2021)
- [Inequality Constrained Stochastic Nonlinear Optimization via Active-Set Sequential Quadratic Programming](https://mlsanity.com/paper/2109.11502/) (2021)
- [Cooper: A Library for Constrained Optimization in Deep Learning](https://mlsanity.com/paper/2504.01212/) (2025)
- [On the Iteration Complexity of Smoothed Proximal ALM for Nonconvex Optimization Problem with Convex Constraints](https://mlsanity.com/paper/2207.06304/) (2022)
- [Single-loop Algorithms for Stochastic Non-convex Optimization with Weakly-Convex Constraints](https://mlsanity.com/paper/2504.15243/) (2025)
- [Position: Adopt Constraints Over Fixed Penalties in Deep Learning](https://mlsanity.com/paper/2505.20628/) (2025)
- [Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs](https://mlsanity.com/paper/2408.15047/) (2024)
- [Stochastic Approximation for Expectation Objective and Expectation Inequality-Constrained Nonconvex Optimization](https://mlsanity.com/paper/2307.02943/) (2023)
- [Deep Learning as the Disciplined Construction of Tame Objects](https://mlsanity.com/paper/2509.18025/) (2025)
- [Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity](https://mlsanity.com/paper/2605.07097/) (2026)
- [A Trust-Region Interior-Point Stochastic Sequential Quadratic Programming Method](https://mlsanity.com/paper/2603.10230/) (2026)
- [humancompatible.train: Implementing Optimization Algorithms for Stochastically-Constrained Stochastic Optimization Problems](https://mlsanity.com/paper/2509.21254/) (2025)

## Abstract (arXiv)

Constrained Machine Learning (CML) enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain knowledge into statistical models. In this work, we introduce the Stochastic Penalty-Barrier Method (SPBM) for CML problems. SPBM extends classical penalty and barrier methods by incorporating an exponential averaging of the dual variables, a stabilized penalty schedule, and the Moreau envelope to handle non-smoothness. We analyze the bias that mini-batching introduces in the barrier function and show that the feasible set of the resulting transformed problem is contained within the original one. We compare SPBM with CML baselines across multiple fairness and physics informed neural networks experiments. We find that SPBM is competitive with state-of-the-art methods. We also observe, on our fairness-based computational benchmark, that the per-epoch runtime of CML methods is largely independent of the number of constraints, and within 1.3× of the per-epoch runtime of regularized Adam, for a number of constraints ranging from 90 to 9900.