# Uncertainty-Aware Performance Prediction for Highly Configurable Software Systems via Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2212.13359 · HTML: https://mlsanity.com/paper/2212.13359/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 27 Dec 2022.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Constrained Efficient Global Optimization of Expensive Black-box Functions](https://mlsanity.com/paper/2211.00162/) (2022)
- [Finding Faster Configurations using FLASH](https://mlsanity.com/paper/1801.02175/) (2018)
- [Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis](https://mlsanity.com/paper/1709.02280/) (2017)
- [White-Box Analysis over Machine Learning: Modeling Performance of Configurable Systems](https://mlsanity.com/paper/2101.05362/) (2021)
- [White-Box Performance-Influence Models: A Profiling and Learning Approach](https://mlsanity.com/paper/2102.06395/) (2021)
- [The Interaction between Inputs and Configurations fed to Software Systems: an Empirical Study](https://mlsanity.com/paper/2112.07279/) (2021)
- [On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support](https://mlsanity.com/paper/2203.10356/) (2022)

## Abstract (arXiv)

Configurable software systems are employed in many important application domains. Understanding the performance of the systems under all configurations is critical to prevent potential performance issues caused by misconfiguration. However, as the number of configurations can be prohibitively large, it is not possible to measure the system performance under all configurations. Thus, a common approach is to build a prediction model from a limited measurement data to predict the performance of all configurations as scalar values. However, it has been pointed out that there are different sources of uncertainty coming from the data collection or the modeling process, which can make the scalar predictions not certainly accurate. To address this problem, we propose a Bayesian deep learning based method, namely BDLPerf, that can incorporate uncertainty into the prediction model. BDLPerf can provide both scalar predictions for configurations' performance and the corresponding confidence intervals of these scalar predictions. We also develop a novel uncertainty calibration technique to ensure the reliability of the confidence intervals generated by a Bayesian prediction model. Finally, we suggest an efficient hyperparameter tuning technique so as to train the prediction model within a reasonable amount of time whilst achieving high accuracy. Our experimental results on 10 real-world systems show that BDLPerf achieves higher accuracy than existing approaches, in both scalar performance prediction and confidence interval estimation.