# Hardness, Structural Knowledge, and Opportunity: An Analytical Framework for Modular Performance Modeling

arXiv: https://arxiv.org/abs/2509.11000 · HTML: https://mlsanity.com/paper/2509.11000/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 93 references that are in the map.
- First posted on arXiv 13 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification](https://mlsanity.com/paper/2011.09588/) (2020)
- [Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis](https://mlsanity.com/paper/1709.02280/) (2017)
- [The Sample Complexity of Learning Linear Predictors with the Squared Loss](https://mlsanity.com/paper/1406.5143/) (2014)
- [ytopt: Autotuning Scientific Applications for Energy Efficiency at Large Scales](https://mlsanity.com/paper/2303.16245/) (2023)
- [Unicorn: Reasoning about Configurable System Performance through the lens of Causality](https://mlsanity.com/paper/2201.08413/) (2022)
- [PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning](https://mlsanity.com/paper/2507.05995/) (2025)
- [White-Box Analysis over Machine Learning: Modeling Performance of Configurable Systems](https://mlsanity.com/paper/2101.05362/) (2021)
- [Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes](https://mlsanity.com/paper/2203.15988/) (2022)
- [White-Box Performance-Influence Models: A Profiling and Learning Approach](https://mlsanity.com/paper/2102.06395/) (2021)
- [Using Bad Learners to find Good Configurations](https://mlsanity.com/paper/1702.05701/) (2017)
- [A Tight Excess Risk Bound via a Unified PAC-Bayesian-Rademacher-Shtarkov-MDL Complexity](https://mlsanity.com/paper/1710.07732/) (2017)
- [Average Stability is Invariant to Data Preconditioning. Implications to Exp-concave Empirical Risk Minimization](https://mlsanity.com/paper/1601.04011/) (2016)
- [On the Impact of Applying Machine Learning in the Decision-Making of Self-Adaptive Systems](https://mlsanity.com/paper/2103.10194/) (2021)
- [CAMEO: A Causal Transfer Learning Approach for Performance Optimization of Configurable Computer Systems](https://mlsanity.com/paper/2306.07888/) (2023)
- [Predicting Performance of Software Configurations: There is no Silver Bullet](https://mlsanity.com/paper/1911.12643/) (2019)
- [Exploring Local Norms in Exp-concave Statistical Learning](https://mlsanity.com/paper/2302.10726/) (2023)
- [iGen: Dynamic Interaction Inference for Configurable Software](https://mlsanity.com/paper/1903.12247/) (2019)
- [SEAByTE: A Self-adaptive Micro-service System Artifact for Automating A/B Testing](https://mlsanity.com/paper/2204.01343/) (2022)
- [Reducing Large Adaptation Spaces in Self-Adaptive Systems Using Machine Learning](https://mlsanity.com/paper/2306.01404/) (2023)
- [GroupTuner: Efficient Group-Aware Compiler Auto-Tuning](https://mlsanity.com/paper/2505.08598/) (2025)

## Abstract (arXiv)

Performance-influence models are beneficial for understanding how configurations affect system performance, but their creation is challenging due to the exponential growth of configuration spaces. While gray-box approaches leverage selective "structural knowledge" (like the module execution graph of the system) to improve modeling, the relationship between this knowledge, a system's characteristics (we call them "structural aspects"), and potential model improvements is not well understood. This paper addresses this gap by formally investigating how variations in structural aspects (e.g., the number of modules and options per module) and the level of structural knowledge impact the creation of "opportunities" for improved "modular performance modeling". We introduce and quantify the concept of modeling "hardness", defined as the inherent difficulty of performance modeling. Through controlled experiments with synthetic system models, we establish an "analytical matrix" to measure these concepts. Our findings show that modeling hardness is primarily driven by the number of modules and configuration options per module. More importantly, we demonstrate that both higher levels of structural knowledge and increased modeling hardness significantly enhance the opportunity for improvement. The impact of these factors varies by performance metric; for ranking accuracy (e.g., in debugging task), structural knowledge is more dominant, while for prediction accuracy (e.g., in resource management task), hardness plays a stronger role. These results provide actionable insights for system designers, guiding them to strategically allocate time and select appropriate modeling approaches based on a system's characteristics and a given task's objectives.