# The Impact of Automated Parameter Optimization on Defect Prediction Models

arXiv: https://arxiv.org/abs/1801.10270 · HTML: https://mlsanity.com/paper/1801.10270/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2018.

## Built on by (most cited first, 30 of 38)

- [The impact of feature importance methods on the interpretation of defect classifiers](https://mlsanity.com/paper/2202.02389/) (2022)
- [Towards a consistent interpretation of AIOps models](https://mlsanity.com/paper/2202.02298/) (2022)
- [Can I Solve It? Identifying APIs Required to Complete OSS Task](https://mlsanity.com/paper/2103.12653/) (2021)
- [On the Effectiveness of Log Representation for Log-based Anomaly Detection](https://mlsanity.com/paper/2308.08736/) (2023)
- [SQAPlanner: Generating Data-Informed Software Quality Improvement Plans](https://mlsanity.com/paper/2102.09687/) (2021)
- [Fairness Concerns in App Reviews: A Study on AI-based Mobile Apps](https://mlsanity.com/paper/2401.08097/) (2024)
- [Predicting the First Response Latency of Maintainers and Contributors in Pull Requests](https://mlsanity.com/paper/2311.07786/) (2023)
- [Tag that issue: Applying API-domain labels in issue tracking systems](https://mlsanity.com/paper/2304.02877/) (2023)
- [LicenseGPT: A Fine-tuned Foundation Model for Publicly Available Dataset License Compliance](https://mlsanity.com/paper/2501.00106/) (2024)
- [Evaluating the Use of LLMs for Documentation to Code Traceability](https://mlsanity.com/paper/2506.16440/) (2025)
- [CoTune: Co-evolutionary Configuration Tuning](https://mlsanity.com/paper/2509.24694/) (2025)
- [Dazzle: Using Optimized Generative Adversarial Networks to Address Security Data Class Imbalance Issue](https://mlsanity.com/paper/2203.11410/) (2022)
- [When Less is More: On the Value of "Co-training" for Semi-Supervised Software Defect Predictors](https://mlsanity.com/paper/2211.05920/) (2022)
- [Evaluating Useful Surrogate Models for Configuration Tuning Beyond Accuracy: A Fitness Landscape Analysis Perspective](https://mlsanity.com/paper/2509.21945/) (2025)
- [Moving from Cross-Project Defect Prediction to Heterogeneous Defect Prediction: A Partial Replication Study](https://mlsanity.com/paper/2103.03490/) (2021)
- [A Comparison of Similarity Based Instance Selection Methods for Cross Project Defect Prediction](https://mlsanity.com/paper/2104.01024/) (2021)
- [Using Voice and Biofeedback to Predict User Engagement during Product Feedback Interviews](https://mlsanity.com/paper/2104.02410/) (2021)
- [Assessing the Early Bird Heuristic (for Predicting Project Quality)](https://mlsanity.com/paper/2105.11082/) (2021)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Predicting Code Review Completion Time in Modern Code Review](https://mlsanity.com/paper/2109.15141/) (2021)
- [A Mining Software Repository Extended Cookbook: Lessons learned from a literature review](https://mlsanity.com/paper/2110.04095/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Do Not Take It for Granted: Comparing Open-Source Libraries for Software Development Effort Estimation](https://mlsanity.com/paper/2207.01705/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)
- [Text Tells the Cost: Predicting and Analyzing Repayment Effort of Self-Admitted Technical Debt](https://mlsanity.com/paper/2309.06020/) (2023)

## Abstract (arXiv)

Defect prediction models---classifiers that identify defect-prone software modules---have configurable parameters that control their characteristics (e.g., the number of trees in a random forest). Recent studies show that these classifiers underperform when default settings are used. In this paper, we study the impact of automated parameter optimization on defect prediction models. Through a case study of 18 datasets, we find that automated parameter optimization: (1) improves AUC performance by up to 40 percentage points; (2) yields classifiers that are at least as stable as those trained using default settings; (3) substantially shifts the importance ranking of variables, with as few as 28% of the top-ranked variables in optimized classifiers also being top-ranked in non-optimized classifiers; (4) yields optimized settings for 17 of the 20 most sensitive parameters that transfer among datasets without a statistically significant drop in performance; and (5) adds less than 30 minutes of additional computation to 12 of the 26 studied classification techniques. While widely-used classification techniques like random forest and support vector machines are not optimization-sensitive, traditionally overlooked techniques like C5.0 and neural networks can actually outperform widely-used techniques after optimization is applied. This highlights the importance of exploring the parameter space when using parameter-sensitive classification techniques.