# Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization

arXiv: https://arxiv.org/abs/1611.02120 · HTML: https://mlsanity.com/paper/1611.02120/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2016.

## Built on by (most cited first, 10 of 10)

- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [A survey on multi-objective hyperparameter optimization algorithms for Machine Learning](https://mlsanity.com/paper/2111.13755/) (2021)
- [U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search](https://mlsanity.com/paper/2203.12412/) (2022)
- [A Survey on Multi-Objective based Parameter Optimization for Deep Learning](https://mlsanity.com/paper/2305.10014/) (2023)
- [HyperTuner: A Cross-Layer Multi-Objective Hyperparameter Auto-Tuning Framework for Data Analytic Services](https://mlsanity.com/paper/2304.10051/) (2023)
- [Cluster Workload Allocation: A Predictive Approach Leveraging Machine Learning Efficiency](https://mlsanity.com/paper/2509.17695/) (2025)
- [TreeNet: Layered Decision Ensembles](https://mlsanity.com/paper/2510.09654/) (2025)
- [Beyond Foundation Models: Dimension-Aware Neural Architecture Search with Small-Data Representation Models for Cryocooler Lifetime Prediction](https://mlsanity.com/paper/2608.06993/) (2026)
- [Energy-aware frugal Bayesian optimization](https://mlsanity.com/paper/2609.31638/) (2026)

## Abstract (arXiv)

Artificial neural networks have gone through a recent rise in popularity, achieving state-of-the-art results in various fields, including image classification, speech recognition, and automated control. Both the performance and computational complexity of such models are heavily dependant on the design of characteristic hyper-parameters (e.g., number of hidden layers, nodes per layer, or choice of activation functions), which have traditionally been optimized manually. With machine learning penetrating low-power mobile and embedded areas, the need to optimize not only for performance (accuracy), but also for implementation complexity, becomes paramount. In this work, we present a multi-objective design space exploration method that reduces the number of solution networks trained and evaluated through response surface modelling. Given spaces which can easily exceed 1020 solutions, manually designing a near-optimal architecture is unlikely as opportunities to reduce network complexity, while maintaining performance, may be overlooked. This problem is exacerbated by the fact that hyper-parameters which perform well on specific datasets may yield sub-par results on others, and must therefore be designed on a per-application basis. In our work, machine learning is leveraged by training an artificial neural network to predict the performance of future candidate networks. The method is evaluated on the MNIST and CIFAR-10 image datasets, optimizing for both recognition accuracy and computational complexity. Experimental results demonstrate that the proposed method can closely approximate the Pareto-optimal front, while only exploring a small fraction of the design space.