# Differentiable Projection for Constrained Deep Learning

arXiv: https://arxiv.org/abs/2111.10785 · HTML: https://mlsanity.com/paper/2111.10785/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 21 Nov 2021.

## Builds on (most cited first, 14 of 14)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Constrained Convolutional Neural Networks for Weakly Supervised Segmentation](https://mlsanity.com/paper/1506.03648/) (2015)
- [Learning to Optimize: Training Deep Neural Networks for Wireless Resource Management](https://mlsanity.com/paper/1705.09412/) (2017)
- [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021)
- [Fully Convolutional Multi-Class Multiple Instance Learning](https://mlsanity.com/paper/1412.7144/) (2014)
- [CapProNet: Deep Feature Learning via Orthogonal Projections onto Capsule Subspaces](https://mlsanity.com/paper/1805.07621/) (2018)
- [From Image-level to Pixel-level Labeling with Convolutional Networks](https://mlsanity.com/paper/1411.6228/) (2014)
- [A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property](https://mlsanity.com/paper/1311.6091/) (2013)
- [Learning Convolutional Neural Networks using Hybrid Orthogonal Projection and Estimation](https://mlsanity.com/paper/1606.05929/) (2016)
- [A generalized projection-based scheme for solving convex constrained optimization problems](https://mlsanity.com/paper/1803.05617/) (2018)
- [Structured Directional Pruning via Perturbation Orthogonal Projection](https://mlsanity.com/paper/2107.05328/) (2021)

## Built on by (most cited first, 4 of 4)

- [HardNet: Hard-Constrained Neural Networks with Universal Approximation Guarantees](https://mlsanity.com/paper/2410.10807/) (2024)
- [FSNet: Feasibility-Seeking Neural Network for Constrained Optimization with Guarantees](https://mlsanity.com/paper/2506.00362/) (2025)
- [NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning](https://mlsanity.com/paper/2210.00973/) (2022)
- [ORACLE: Optimizer-Relative Alignment for Constrained LEarning](https://mlsanity.com/paper/2610.09040/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have achieved extraordinary performance in solving different tasks in various fields. However, the conventional DNN model is steadily approaching the ground-truth value through loss backpropagation. In some applications, some prior knowledge could be easily obtained, such as constraints which the ground truth observation follows. Here, we try to give a general approach to incorporate information from these constraints to enhance the performance of the DNNs. Theoretically, we could formulate these kinds of problems as constrained optimization problems that KKT conditions could solve. In this paper, we propose to use a differentiable projection layer in DNN instead of directly solving time-consuming KKT conditions. The proposed projection method is differentiable, and no heavy computation is required. Finally, we also conducted some experiments using a randomly generated synthetic dataset and image segmentation task using the PASCAL VOC dataset to evaluate the performance of the proposed projection method. Experimental results show that the projection method is sufficient and outperforms baseline methods.