# Optimizing over an ensemble of neural networks

arXiv: https://arxiv.org/abs/2112.07007 · HTML: https://mlsanity.com/paper/2112.07007/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 51 references that are in the map.
- First posted on arXiv 13 Dec 2021.

## Builds on (most cited first, 20 of 20)

- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Bounding and Counting Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1711.02114/) (2017)
- [From Predictive to Prescriptive Analytics](https://mlsanity.com/paper/1402.5481/) (2014)
- [An approach to reachability analysis for feed-forward ReLU neural networks](https://mlsanity.com/paper/1706.07351/) (2017)
- [Strong mixed-integer programming formulations for trained neural networks](https://mlsanity.com/paper/1811.08359/) (2018)
- [Maximum Resilience of Artificial Neural Networks](https://mlsanity.com/paper/1705.01040/) (2017)
- [A Literature Survey of Benchmark Functions For Global Optimization Problems](https://mlsanity.com/paper/1308.4008/) (2013)
- [Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability](https://mlsanity.com/paper/1809.03008/) (2018)
- [Ensemble Sampling](https://mlsanity.com/paper/1705.07347/) (2017)
- [Partition-based formulations for mixed-integer optimization of trained ReLU neural networks](https://mlsanity.com/paper/2102.04373/) (2021)
- [Lossless Compression of Deep Neural Networks](https://mlsanity.com/paper/2001.00218/) (2020)
- [Global Deterministic Optimization with Artificial Neural Networks Embedded](https://mlsanity.com/paper/1801.07114/) (2018)
- [ReLU Networks as Surrogate Models in Mixed-Integer Linear Programs](https://mlsanity.com/paper/1907.03140/) (2019)
- [Mixed-Integer Optimization with Constraint Learning](https://mlsanity.com/paper/2111.04469/) (2021)
- [JANOS: An Integrated Predictive and Prescriptive Modeling Framework](https://mlsanity.com/paper/1911.09461/) (2019)
- [Optimization of Tree Ensembles](https://mlsanity.com/paper/1705.10883/) (2017)
- [Model Distillation for Revenue Optimization: Interpretable Personalized Pricing](https://mlsanity.com/paper/2007.01903/) (2020)
- [Mixed-Integer Convex Nonlinear Optimization with Gradient-Boosted Trees Embedded](https://mlsanity.com/paper/1803.00952/) (2018)
- [ENTMOOT: A Framework for Optimization over Ensemble Tree Models](https://mlsanity.com/paper/2003.04774/) (2020)
- [Learning optimization models in the presence of unknown relations](https://mlsanity.com/paper/1401.1061/) (2014)

## Built on by (most cited first, 1 of 1)

- [Subsampled Ensemble Can Improve Generalization Tail Exponentially](https://mlsanity.com/paper/2405.14741/) (2024)

## Abstract (arXiv)

We study optimization problems where the objective function is modeled through feedforward neural networks with rectified linear unit (ReLU) activation. Recent literature has explored the use of a single neural network to model either uncertain or complex elements within an objective function. However, it is well known that ensembles of neural networks produce more stable predictions and have better generalizability than models with single neural networks, which motivates the investigation of ensembles of neural networks rather than single neural networks in decision-making pipelines. We study how to incorporate a neural network ensemble as the objective function of an optimization model and explore computational approaches for the ensuing problem. We present a mixed-integer linear program based on existing popular big-M formulations for optimizing over a single neural network. We develop a two-phase approach for our model that combines preprocessing procedures to tighten bounds for critical neurons in the neural networks with a Lagrangian relaxation-based branch-and-bound approach. Experimental evaluations of our solution methods suggest that using ensembles of neural networks yields more stable and higher quality solutions, compared to single neural networks, and that our optimization algorithm outperforms (the adaption of) a state-of-the-art approach in terms of computational time and optimality gaps.