# A multi-objective perspective on jointly tuning hardware and hyperparameters

arXiv: https://arxiv.org/abs/2106.05680 · HTML: https://mlsanity.com/paper/2106.05680/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 12 of 12)

- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution](https://mlsanity.com/paper/1804.09081/) (2018)
- [HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark](https://mlsanity.com/paper/2103.10584/) (2021)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization](https://mlsanity.com/paper/2006.04655/) (2020)
- [Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization](https://mlsanity.com/paper/1905.04970/) (2019)
- [Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning](https://mlsanity.com/paper/1909.12552/) (2019)
- [A Bandit-Based Algorithm for Fairness-Aware Hyperparameter Optimization](https://mlsanity.com/paper/2010.03665/) (2020)

## Built on by (most cited first, 12 of 12)

- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Multi-objective Asynchronous Successive Halving](https://mlsanity.com/paper/2106.12639/) (2021)
- [Tuning LLM Judge Design Decisions for 1/1000 of the Cost](https://mlsanity.com/paper/2501.17178/) (2025)
- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [Multi-Objective Model Selection for Time Series Forecasting](https://mlsanity.com/paper/2202.08485/) (2022)
- [Multi-Objective Population Based Training](https://mlsanity.com/paper/2306.01436/) (2023)
- [Tackling Neural Architecture Search With Quality Diversity Optimization](https://mlsanity.com/paper/2208.00204/) (2022)
- [MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization](https://mlsanity.com/paper/2305.04502/) (2023)
- [Multi-objective Differentiable Neural Architecture Search](https://mlsanity.com/paper/2402.18213/) (2024)
- [Multi-objective Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2511.08371/) (2025)
- [Automated Computational Energy Minimization of ML Algorithms using Constrained Bayesian Optimization](https://mlsanity.com/paper/2407.05788/) (2024)
- [Efficient Hyperparameter Search for Non-Stationary Model Training](https://mlsanity.com/paper/2512.01258/) (2025)

## Abstract (arXiv)

In addition to the best model architecture and hyperparameters, a full AutoML solution requires selecting appropriate hardware automatically. This can be framed as a multi-objective optimization problem: there is not a single best hardware configuration but a set of optimal ones achieving different trade-offs between cost and runtime. In practice, some choices may be overly costly or take days to train. To lift this burden, we adopt a multi-objective approach that selects and adapts the hardware configuration automatically alongside neural architectures and their hyperparameters. Our method builds on Hyperband and extends it in two ways. First, we replace the stopping rule used in Hyperband by a non-dominated sorting rule to preemptively stop unpromising configurations. Second, we leverage hyperparameter evaluations from related tasks via transfer learning by building a probabilistic estimate of the Pareto front that finds promising configurations more efficiently than random search. We show in extensive NAS and HPO experiments that both ingredients bring significant speed-ups and cost savings, with little to no impact on accuracy. In three benchmarks where hardware is selected in addition to hyperparameters, we obtain runtime and cost reductions of at least 5.8x and 8.8x, respectively. Furthermore, when applying our multi-objective method to the tuning of hyperparameters only, we obtain a 10% improvement in runtime while maintaining the same accuracy on two popular NAS benchmarks.