# Hyperparameter optimization of data-driven AI models on HPC systems

arXiv: https://arxiv.org/abs/2203.01112 · HTML: https://mlsanity.com/paper/2203.01112/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 8 of 8)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [MLPF: Efficient machine-learned particle-flow reconstruction using graph neural networks](https://mlsanity.com/paper/2101.08578/) (2021)
- [Machine Learning for Particle Flow Reconstruction at CMS](https://mlsanity.com/paper/2203.00330/) (2022)

## Built on by (most cited first, 3 of 3)

- [Progress towards an improved particle flow algorithm at CMS with machine learning](https://mlsanity.com/paper/2303.17657/) (2023)
- [Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors](https://mlsanity.com/paper/2309.06782/) (2023)
- [CNN-based local features for navigation near an asteroid](https://mlsanity.com/paper/2309.11156/) (2023)

## Abstract (arXiv)

In the European Center of Excellence in Exascale computing "Research on AI- and Simulation-Based Engineering at Exascale" (CoE RAISE), researchers develop novel, scalable AI technologies towards Exascale. This work exercises High Performance Computing resources to perform large-scale hyperparameter optimization using distributed training on multiple compute nodes. This is part of RAISE's work on data-driven use cases which leverages AI- and HPC cross-methods developed within the project. In response to the demand for parallelizable and resource efficient hyperparameter optimization methods, advanced hyperparameter search algorithms are benchmarked and compared. The evaluated algorithms, including Random Search, Hyperband and ASHA, are tested and compared in terms of both accuracy and accuracy per compute resources spent. As an example use case, a graph neural network model known as MLPF, developed for the task of Machine-Learned Particle-Flow reconstruction in High Energy Physics, acts as the base model for optimization. Results show that hyperparameter optimization significantly increased the performance of MLPF and that this would not have been possible without access to large-scale High Performance Computing resources. It is also shown that, in the case of MLPF, the ASHA algorithm in combination with Bayesian optimization gives the largest performance increase per compute resources spent out of the investigated algorithms.