# Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks

arXiv: https://arxiv.org/abs/2110.11382 · HTML: https://mlsanity.com/paper/2110.11382/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 48 references that are in the map.
- First posted on arXiv 21 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Adversarial Examples: Attacks and Defenses for Deep Learning](https://mlsanity.com/paper/1712.07107/) (2017)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [Strong mixed-integer programming formulations for trained neural networks](https://mlsanity.com/paper/1811.08359/) (2018)
- [Precise Tradeoffs in Adversarial Training for Linear Regression](https://mlsanity.com/paper/2002.10477/) (2020)
- [Provable tradeoffs in adversarially robust classification](https://mlsanity.com/paper/2006.05161/) (2020)
- [Efficient Exact Verification of Binarized Neural Networks](https://mlsanity.com/paper/2005.03597/) (2020)
- [Bitwise Neural Networks](https://mlsanity.com/paper/1601.06071/) (2016)
- [Learning Optimal Power Flow: Worst-Case Guarantees for Neural Networks](https://mlsanity.com/paper/2006.11029/) (2020)
- [Verification of Neural Network Behaviour: Formal Guarantees for Power System Applications](https://mlsanity.com/paper/1910.01624/) (2019)
- [Combinatorial Attacks on Binarized Neural Networks](https://mlsanity.com/paper/1810.03538/) (2018)
- [Piecewise convexity of artificial neural networks](https://mlsanity.com/paper/1607.04917/) (2016)
- [An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions](https://mlsanity.com/paper/2007.03326/) (2020)
- [Data-Driven Robust Optimization using Unsupervised Deep Learning](https://mlsanity.com/paper/2011.09769/) (2020)

## Built on by (most cited first, 4 of 4)

- [Optimization Over Trained Neural Networks: Taking a Relaxing Walk](https://mlsanity.com/paper/2401.03451/) (2024)
- [A Scalable, Interpretable, Verifiable & Differentiable Logic Gate Convolutional Neural Network Architecture From Truth Tables](https://mlsanity.com/paper/2208.08609/) (2022)
- [A Mixed-Integer Programming Approach to Training Dense Neural Networks](https://mlsanity.com/paper/2201.00723/) (2022)
- [Multi-Objective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networks](https://mlsanity.com/paper/2212.03659/) (2022)

## Abstract (arXiv)

Compared to classical deep neural networks its binarized versions can be useful for applications on resource-limited devices due to their reduction in memory consumption and computational demands. In this work we study deep neural networks with binary activation functions and continuous or integer weights (BDNN). We show that the BDNN can be reformulated as a mixed-integer linear program with bounded weight space which can be solved to global optimality by classical mixed-integer programming solvers. Additionally, a local search heuristic is presented to calculate locally optimal networks. Furthermore to improve efficiency we present an iterative data-splitting heuristic which iteratively splits the training set into smaller subsets by using the k-mean method. Afterwards all data points in a given subset are forced to follow the same activation pattern, which leads to a much smaller number of integer variables in the mixed-integer programming formulation and therefore to computational improvements. Finally for the first time a robust model is presented which enforces robustness of the BDNN during training. All methods are tested on random and real datasets and our results indicate that all models can often compete with or even outperform classical DNNs on small network architectures confirming the viability for applications having restricted memory or computing power.