# Is Model Instability just Noise to be Tolerated or a Property that can be Managed?

arXiv: https://arxiv.org/abs/2607.10420 · HTML: https://mlsanity.com/paper/2607.10420/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 11 Jul 2026.

## Builds on (most cited first, 28 of 28)

- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [On the Existence of Simpler Machine Learning Models](https://mlsanity.com/paper/1908.01755/) (2019)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [A Path to Simpler Models Starts With Noise](https://mlsanity.com/paper/2310.19726/) (2023)
- [Finding Faster Configurations using FLASH](https://mlsanity.com/paper/1801.02175/) (2018)
- [The Current Challenges of Software Engineering in the Era of Large Language Models](https://mlsanity.com/paper/2412.14554/) (2024)
- [Tuning for Software Analytics: is it Really Necessary?](https://mlsanity.com/paper/1609.01759/) (2016)
- [Detecting False Alarms from Automatic Static Analysis Tools: How Far are We?](https://mlsanity.com/paper/2202.05982/) (2022)
- [What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)](https://mlsanity.com/paper/1608.08176/) (2016)
- [PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning](https://mlsanity.com/paper/2507.05995/) (2025)
- [How to "DODGE" Complex Software Analytics?](https://mlsanity.com/paper/1902.01838/) (2019)
- [Identifying Self-Admitted Technical Debts with Jitterbug: A Two-step Approach](https://mlsanity.com/paper/2002.11049/) (2020)
- [Bellwethers: A Baseline Method For Transfer Learning](https://mlsanity.com/paper/1703.06218/) (2017)
- [FRUGAL: Unlocking SSL for Software Analytics](https://mlsanity.com/paper/2108.09847/) (2021)
- [Shaky Structures: The Wobbly World of Causal Graphs in Software Analytics](https://mlsanity.com/paper/2505.12554/) (2025)
- [MOOT: a Repository of Many Multi-Objective Optimization Tasks](https://mlsanity.com/paper/2511.16882/) (2025)
- [Software Analytics to Software Domains: A Systematic Literature Review](https://mlsanity.com/paper/1511.04109/) (2015)
- [Impact of Software Engineering Research in Practice: A Patent and Author Survey Analysis](https://mlsanity.com/paper/2204.03366/) (2022)
- [Analyzing the Instability of Large Language Models in Automated Bug Injection and Correction](https://mlsanity.com/paper/2509.06429/) (2025)
- [Minimal Data, Maximum Clarity: A Heuristic for Explaining Optimization](https://mlsanity.com/paper/2509.08667/) (2025)
- [Oops!... I did it again. Analysing and Handling Conclusion (In-)Stability in Socio-Technical Software Engineering](https://mlsanity.com/paper/2510.06844/) (2025)
- [How Low Can You Go? The Data-Light SE Challenge](https://mlsanity.com/paper/2512.13524/) (2025)
- [Can Causality Cure Confusion Caused By Correlation (in Software Analytics)?](https://mlsanity.com/paper/2602.16091/) (2026)
- [Zoom, Don't Wander: Why Regional Search Outperforms Pareto Reasoning and Global Optimization in Budget-Constrained SBSE](https://mlsanity.com/paper/2605.09658/) (2026)
- [Can AI be Easy? Lessons Learned from the EZR.py Toolkit](https://mlsanity.com/paper/2606.03640/) (2026)

## Abstract (arXiv)

In software analytics, rerunning the same analysis twice often yields different models and conclusions. This reduces trust in the model and limits its use. We find that model instability is a major problem. Across 127 multi-objective SE optimization problems (12,700 test cases), repeated runs of a state-of-the-art optimizer agree on only 13.7% of test cases, even under improved settings. We argue that this instability is not merely noise to tolerate, but a property that can be measured and managed. By adjusting how labels are spent, how complex the models become, and how splits are scored, we obtain models that agree 4.8 times as often as the default configuration. The standard deviation of optimization error falls by 22% on average (mean std 17.4 to 13.6), while recommendation quality improves rather than degrades. In terms of quality, the refined settings are statistically top-ranked on 119 of 127 datasets, compared to 74 for the defaults. We then test causal and data-locality interventions and find that they help only partially, suggesting a residual stability floor. Our evidence suggests there are fundamental limits to stability set by the data itself (noise, scarce labels, proxy objectives, and the many near-equivalent models a dataset admits). We conclude that instability should be treated as a standard evaluation axis in SE optimization, which should be routinely measured, reported alongside performance, and used to calibrate trust in any single run. The methods in this paper provide a baseline against which future efforts to reduce SBSE instability can be judged. To support open science, we offer the following reproduction package: https://tinyurl.com/Model-Instability