# Expressing linear equality constraints in feedforward neural networks

arXiv: https://arxiv.org/abs/2211.04395 · HTML: https://mlsanity.com/paper/2211.04395/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 9 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 1 of 1)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)

## Built on by (most cited first, 2 of 2)

- [Learning to Optimize for Mixed-Integer Non-linear Programming with Feasibility Guarantees](https://mlsanity.com/paper/2410.11061/) (2024)
- [A condensing approach to multiple shooting neural ordinary differential equation](https://mlsanity.com/paper/2506.00724/) (2025)

## Abstract (arXiv)

We seek to impose linear, equality constraints in feedforward neural networks. As top layer predictors are usually nonlinear, this is a difficult task if we seek to deploy standard convex optimization methods and strong duality. To overcome this, we introduce a new saddle-point Lagrangian with auxiliary predictor variables on which constraints are imposed. Elimination of the auxiliary variables leads to a dual minimization problem on the Lagrange multipliers introduced to satisfy the linear constraints. This minimization problem is combined with the standard learning problem on the weight matrices. From this theoretical line of development, we obtain the surprising interpretation of Lagrange parameters as additional, penultimate layer hidden units with fixed weights stemming from the constraints. Consequently, standard minimization approaches can be used despite the inclusion of Lagrange parameters -- a very satisfying, albeit unexpected, discovery. Examples ranging from multi-label classification to constrained autoencoders are envisaged in the future. The code has been made available at https://github.com/anandrajan0/smartalec