# Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs

arXiv: https://arxiv.org/abs/2305.08852 · HTML: https://mlsanity.com/paper/2305.08852/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 14 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 4 of 4)

- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Bayesian Optimization in AlphaGo](https://mlsanity.com/paper/1812.06855/) (2018)
- [Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator](https://mlsanity.com/paper/2212.06751/) (2022)
- [MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization](https://mlsanity.com/paper/2305.04502/) (2023)

## Built on by (most cited first, 4 of 4)

- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [MO-CAPO: Multi-Objective Cost-Aware Prompt Optimization](https://mlsanity.com/paper/2605.18869/) (2026)
- [Preference-Optimal Multi-Metric Weighting for Parallel Coordinate Plots](https://mlsanity.com/paper/2507.02905/) (2025)
- [Approximation of Box Decomposition Algorithm for Fast Hypervolume-Based Multi-Objective Optimization](https://mlsanity.com/paper/2512.05825/) (2025)

## Abstract (arXiv)

Hyperparameter optimization is crucial to achieving high performance in deep learning. On top of the performance, other criteria such as inference time or memory requirement often need to be optimized due to some practical reasons. This motivates research on multi-objective optimization (MOO). However, Pareto fronts of MOO methods are often shown without considering the variability caused by random seeds and this makes the performance stability evaluation difficult. Although there is a concept named empirical attainment surface to enable the visualization with uncertainty over multiple runs, there is no major Python package for empirical attainment surface. We, therefore, develop a Python package for this purpose and describe the usage. The package is available at https://github.com/nabenabe0928/empirical-attainment-func.