# Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes

arXiv: https://arxiv.org/abs/2102.12894 · HTML: https://mlsanity.com/paper/2102.12894/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 21 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [Posterior Re-calibration for Imbalanced Datasets](https://mlsanity.com/paper/2010.11820/) (2020)
- [One-Pass AUC Optimization](https://mlsanity.com/paper/1305.1363/) (2013)
- [Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation](https://mlsanity.com/paper/1907.10982/) (2019)
- [MESA: Boost Ensemble Imbalanced Learning with MEta-SAmpler](https://mlsanity.com/paper/2010.08830/) (2020)
- [Application of Decision Rules for Handling Class Imbalance in Semantic Segmentation](https://mlsanity.com/paper/1901.08394/) (2019)
- [Imbalanced Data Learning by Minority Class Augmentation using Capsule Adversarial Networks](https://mlsanity.com/paper/2004.02182/) (2020)
- [MBA: Mini-Batch AUC Optimization](https://mlsanity.com/paper/1805.11221/) (2018)

## Built on by (most cited first, 19 of 19)

- [Training OOD Detectors in their Natural Habitats](https://mlsanity.com/paper/2202.03299/) (2022)
- [A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models](https://mlsanity.com/paper/2312.12730/) (2023)
- [Robust Calibration of Large Vision-Language Adapters](https://mlsanity.com/paper/2407.13588/) (2024)
- [Class Adaptive Network Calibration](https://mlsanity.com/paper/2211.15088/) (2022)
- [Geometry-Informed Neural Networks](https://mlsanity.com/paper/2402.14009/) (2024)
- [Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems](https://mlsanity.com/paper/2110.00604/) (2021)
- [Composite Score for Anomaly Detection in Imbalanced Real-World Industrial Dataset](https://mlsanity.com/paper/2211.15513/) (2022)
- [Expert load matters: operating networks at high accuracy and low manual effort](https://mlsanity.com/paper/2308.05035/) (2023)
- [Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach](https://mlsanity.com/paper/2310.16647/) (2023)
- [WANCO: Weak Adversarial Networks for Constrained Optimization problems](https://mlsanity.com/paper/2407.03647/) (2024)
- [Ranking Regularization for Critical Rare Classes: Minimizing False Positives at a High True Positive Rate](https://mlsanity.com/paper/2304.00049/) (2023)
- [The inexact power augmented Lagrangian method for constrained nonconvex optimization](https://mlsanity.com/paper/2410.20153/) (2024)
- [mPOLICE: Provable Enforcement of Multi-Region Affine Constraints in Deep Neural Networks](https://mlsanity.com/paper/2502.02434/) (2025)
- [What Time Tells Us? An Explorative Study of Time Awareness Learned from Static Images](https://mlsanity.com/paper/2503.17899/) (2025)
- [QuadAttack: A Quadratic Programming Approach to Ordered Top-K Attacks](https://mlsanity.com/paper/2312.11510/) (2023)
- [A Two-Stage Training Method for Modeling Constrained Systems With Neural Networks](https://mlsanity.com/paper/2403.02730/) (2024)
- [Class Adaptive Conformal Training](https://mlsanity.com/paper/2601.09522/) (2026)
- [Sample-wise Constrained Learning via a Sequential Penalty Approach with Applications in Image Processing](https://mlsanity.com/paper/2601.16812/) (2026)
- [Local LMO is Secretly a Projection Method!](https://mlsanity.com/paper/2609.33383/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) are notorious for making more mistakes for the classes that have substantially fewer samples than the others during training. Such class imbalance is ubiquitous in clinical applications and very crucial to handle because the classes with fewer samples most often correspond to critical cases (e.g., cancer) where misclassifications can have severe consequences. Not to miss such cases, binary classifiers need to be operated at high True Positive Rates (TPRs) by setting a higher threshold, but this comes at the cost of very high False Positive Rates (FPRs) for problems with class imbalance. Existing methods for learning under class imbalance most often do not take this into account. We argue that prediction accuracy should be improved by emphasizing reducing FPRs at high TPRs for problems where misclassification of the positive, i.e. critical, class samples are associated with higher cost. To this end, we pose the training of a DNN for binary classification as a constrained optimization problem and introduce a novel constraint that can be used with existing loss functions to enforce maximal area under the ROC curve (AUC) through prioritizing FPR reduction at high TPR. We solve the resulting constrained optimization problem using an Augmented Lagrangian method (ALM). Going beyond binary, we also propose two possible extensions of the proposed constraint for multi-class classification problems. We present experimental results for image-based binary and multi-class classification applications using an in-house medical imaging dataset, CIFAR10, and CIFAR100. Our results demonstrate that the proposed method improves the baselines in majority of the cases by attaining higher accuracy on critical classes while reducing the misclassification rate for the non-critical class samples.