# Feed-Forward Neural Networks as a Mixed-Integer Program

arXiv: https://arxiv.org/abs/2402.06697 · HTML: https://mlsanity.com/paper/2402.06697/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Builds on (most cited first, 23 of 23)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon](https://mlsanity.com/paper/1811.06128/) (2018)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [Algorithms for Verifying Deep Neural Networks](https://mlsanity.com/paper/1903.06758/) (2019)
- [Bounding and Counting Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1711.02114/) (2017)
- [Branch and Bound for Piecewise Linear Neural Network Verification](https://mlsanity.com/paper/1909.06588/) (2019)
- [An approach to reachability analysis for feed-forward ReLU neural networks](https://mlsanity.com/paper/1706.07351/) (2017)
- [Strong mixed-integer programming formulations for trained neural networks](https://mlsanity.com/paper/1811.08359/) (2018)
- [Maximum Resilience of Artificial Neural Networks](https://mlsanity.com/paper/1705.01040/) (2017)
- [Partition-based formulations for mixed-integer optimization of trained ReLU neural networks](https://mlsanity.com/paper/2102.04373/) (2021)
- [Output Range Analysis for Deep Neural Networks](https://mlsanity.com/paper/1709.09130/) (2017)
- [Empirical Bounds on Linear Regions of Deep Rectifier Networks](https://mlsanity.com/paper/1810.03370/) (2018)
- [Optimization Problems for Machine Learning: A Survey](https://mlsanity.com/paper/1901.05331/) (2019)
- [Reinforcement Learning for Joint Optimization of Multiple Rewards](https://mlsanity.com/paper/1909.02940/) (2019)
- [Equivalent and Approximate Transformations of Deep Neural Networks](https://mlsanity.com/paper/1905.11428/) (2019)
- [Between steps: Intermediate relaxations between big-M and convex hull formulations](https://mlsanity.com/paper/2101.12708/) (2021)
- [Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory](https://mlsanity.com/paper/2006.11052/) (2020)
- [Optimization Methods for Supervised Machine Learning: From Linear Models to Deep Learning](https://mlsanity.com/paper/1706.10207/) (2017)
- [Identifying Critical Neurons in ANN Architectures using Mixed Integer Programming](https://mlsanity.com/paper/2002.07259/) (2020)
- [An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions](https://mlsanity.com/paper/2007.03326/) (2020)

## Built on by (most cited first, 4 of 4)

- [SAQNN: Spectral Adaptive Quantum Neural Network as a Universal Approximator](https://mlsanity.com/paper/2602.09718/) (2026)
- [ArmFormer: Lightweight Transformer Architecture for Real-Time Multi-Class Weapon Segmentation and Classification](https://mlsanity.com/paper/2510.16854/) (2025)
- [On the Expressive Power of Weight Quantization in Large Language Models](https://mlsanity.com/paper/2606.22249/) (2026)
- [CG4AI: A Column Generation Framework for Training AI Models Under Constraints](https://mlsanity.com/paper/2608.26375/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) are widely studied in various applications. A DNN consists of layers of neurons that compute affine combinations, apply nonlinear operations, and produce corresponding activations. The rectified linear unit (ReLU) is a typical nonlinear operator, outputting the max of its input and zero. In scenarios like max pooling, where multiple input values are involved, a fixed-parameter DNN can be modeled as a mixed-integer program (MIP). This formulation, with continuous variables representing unit outputs and binary variables for ReLU activation, finds applications across diverse domains. This study explores the formulation of trained ReLU neurons as MIP and applies MIP models for training neural networks (NNs). Specifically, it investigates interactions between MIP techniques and various NN architectures, including binary DNNs (employing step activation functions) and binarized DNNs (with weights and activations limited to -1,0,+1). The research focuses on training and evaluating proposed approaches through experiments on handwritten digit classification models. The comparative study assesses the performance of trained ReLU NNs, shedding light on the effectiveness of MIP formulations in enhancing training processes for NNs.