# Sample-wise Constrained Learning via a Sequential Penalty Approach with Applications in Image Processing

arXiv: https://arxiv.org/abs/2601.16812 · HTML: https://mlsanity.com/paper/2601.16812/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [HiDDeN: Hiding Data With Deep Networks](https://mlsanity.com/paper/1807.09937/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Adversarial Attacks and Defenses in Images, Graphs and Text: A Review](https://mlsanity.com/paper/1909.08072/) (2019)
- [Lagrangian Duality for Constrained Deep Learning](https://mlsanity.com/paper/2001.09394/) (2020)
- [Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition](https://mlsanity.com/paper/1308.6370/) (2013)
- [Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learning](https://mlsanity.com/paper/2402.03917/) (2024)
- [Enhancing Explainability of Neural Networks through Architecture Constraints](https://mlsanity.com/paper/1901.03838/) (2019)
- [Conditional Mutual Information for Disentangled Representations in Reinforcement Learning](https://mlsanity.com/paper/2305.14133/) (2023)
- [Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints](https://mlsanity.com/paper/2208.04425/) (2022)
- [Training neural networks under physical constraints using a stochastic augmented Lagrangian approach](https://mlsanity.com/paper/2009.07330/) (2020)
- [Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes](https://mlsanity.com/paper/2102.12894/) (2021)
- [Stochastic Inexact Augmented Lagrangian Method for Nonconvex Expectation Constrained Optimization](https://mlsanity.com/paper/2212.09513/) (2022)
- [Conditional Mutual Information Constrained Deep Learning for Classification](https://mlsanity.com/paper/2309.09123/) (2023)
- [Penalty Methods with Stochastic Approximation for Stochastic Nonlinear Programming](https://mlsanity.com/paper/1312.2690/) (2013)
- [Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization](https://mlsanity.com/paper/2411.15931/) (2024)
- [Lagrangian dual framework for conservative neural network solutions of kinetic equations](https://mlsanity.com/paper/2106.12147/) (2021)
- [Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach](https://mlsanity.com/paper/2310.16647/) (2023)
- [ASPEN: An Additional Sampling Penalty Method for Finite-Sum Optimization Problems with Nonlinear Equality Constraints](https://mlsanity.com/paper/2508.02299/) (2025)

## Abstract (arXiv)

In many learning tasks, certain requirements on the processing of individual data samples should arguably be formalized as strict constraints in the underlying optimization problem, rather than by means of arbitrary penalties. We show that, in these scenarios, learning can be carried out exploiting a sequential penalty method that allows to properly deal with constraints. The proposed algorithm is shown to possess convergence guarantees under assumptions that are reasonable in deep learning scenarios. Moreover, the results of experiments on image processing tasks show that the method is indeed viable to be used in practice.