# Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?

arXiv: https://arxiv.org/abs/1904.03786 · HTML: https://mlsanity.com/paper/1904.03786/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2019.

## Built on by (most cited first, 10 of 10)

- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Reconsidering CO2 emissions from Computer Vision](https://mlsanity.com/paper/2104.08702/) (2021)
- [TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets](https://mlsanity.com/paper/2204.07615/) (2022)
- [Contrastive Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2102.10557/) (2021)
- [Search-time Efficient Device Constraints-Aware Neural Architecture Search](https://mlsanity.com/paper/2307.04443/) (2023)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [SplitBeam: Effective and Efficient Beamforming in Wi-Fi Networks Through Split Computing](https://mlsanity.com/paper/2310.08656/) (2023)
- [Fusing Conditional Submodular GAN and Programmatic Weak Supervision](https://mlsanity.com/paper/2312.10366/) (2023)
- [A Survey on Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2307.09099/) (2023)

## Abstract (arXiv)

The design of neural network architectures is frequently either based on human expertise using trial/error and empirical feedback or tackled via large scale reinforcement learning strategies performed over distinct discrete architecture choices. In the latter case, the optimization is often non-differentiable and also not very amenable to derivative-free optimization methods. Most methods in use today require sizable computational resources. And if we want networks that additionally satisfy resource constraints, the above challenges are exacerbated because the search must now balance accuracy with certain budget constraints on resources. We formulate this problem as the optimization of a set function -- we find that the empirical behavior of this set function often (but not always) satisfies marginal gain and monotonicity principles -- properties central to the idea of submodularity. Based on this observation, we adapt algorithms within discrete optimization to obtain heuristic schemes for neural network architecture search, where we have resource constraints on the architecture. This simple scheme when applied on CIFAR-100 and ImageNet, identifies resource-constrained architectures with quantifiably better performance than current state-of-the-art models designed for mobile devices. Specifically, we find high-performing architectures with fewer parameters and computations by a search method that is much faster.