# Constrained Empirical Risk Minimization: Theory and Practice

arXiv: https://arxiv.org/abs/2302.04729 · HTML: https://mlsanity.com/paper/2302.04729/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Input Convex Neural Networks](https://mlsanity.com/paper/1609.07152/) (2016)
- [DC3: A learning method for optimization with hard constraints](https://mlsanity.com/paper/2104.12225/) (2021)
- [Constrained Convolutional Neural Networks for Weakly Supervised Segmentation](https://mlsanity.com/paper/1506.03648/) (2015)
- [Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks](https://mlsanity.com/paper/2001.05992/) (2020)
- [Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds](https://mlsanity.com/paper/2106.06020/) (2021)
- [Neural Conservation Laws: A Divergence-Free Perspective](https://mlsanity.com/paper/2210.01741/) (2022)
- [Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds](https://mlsanity.com/paper/1605.07147/) (2016)
- [Imposing Hard Constraints on Deep Networks: Promises and Limitations](https://mlsanity.com/paper/1706.02025/) (2017)
- [POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networks](https://mlsanity.com/paper/2211.01340/) (2022)
- [Better Training using Weight-Constrained Stochastic Dynamics](https://mlsanity.com/paper/2106.10704/) (2021)
- [Constraint-Based Regularization of Neural Networks](https://mlsanity.com/paper/2006.10114/) (2020)

## Built on by (most cited first, 1 of 1)

- [Myopically Verifiable Probabilistic Certificates for Safe Control and Learning](https://mlsanity.com/paper/2404.16883/) (2024)

## Abstract (arXiv)

Deep Neural Networks (DNNs) are widely used for their ability to effectively approximate large classes of functions. This flexibility, however, makes the strict enforcement of constraints on DNNs an open problem. Here we present a framework that, under mild assumptions, allows the exact enforcement of constraints on parameterized sets of functions such as DNNs. Instead of imposing "soft'' constraints via additional terms in the loss, we restrict (a subset of) the DNN parameters to a submanifold on which the constraints are satisfied exactly throughout the entire training procedure. We focus on constraints that are outside the scope of equivariant networks used in Geometric Deep Learning. As a major example of the framework, we restrict filters of a Convolutional Neural Network (CNN) to be wavelets, and apply these wavelet networks to the task of contour prediction in the medical domain.