# Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health

arXiv: https://arxiv.org/abs/2301.06577 · HTML: https://mlsanity.com/paper/2301.06577/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 77 references that are in the map.
- First posted on arXiv 16 Jan 2023.

## Builds on (most cited first, 21 of 21)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Finding Faster Configurations using FLASH](https://mlsanity.com/paper/1801.02175/) (2018)
- [Tuning for Software Analytics: is it Really Necessary?](https://mlsanity.com/paper/1609.01759/) (2016)
- [Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)](https://mlsanity.com/paper/1705.03697/) (2017)
- [Hyperparameter Optimization for Effort Estimation](https://mlsanity.com/paper/1805.00336/) (2018)
- [Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions](https://mlsanity.com/paper/1901.02302/) (2019)
- [How to "DODGE" Complex Software Analytics?](https://mlsanity.com/paper/1902.01838/) (2019)
- [Identifying Self-Admitted Technical Debts with Jitterbug: A Two-step Approach](https://mlsanity.com/paper/2002.11049/) (2020)
- ["Sampling"' as a Baseline Optimizer for Search-based Software Engineering](https://mlsanity.com/paper/1608.07617/) (2016)
- [Evaluating prediction systems in software project estimation](https://mlsanity.com/paper/2101.05426/) (2021)
- [FRUGAL: Unlocking SSL for Software Analytics](https://mlsanity.com/paper/2108.09847/) (2021)
- [Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?](https://mlsanity.com/paper/2008.07334/) (2020)
- [DebtFree: Minimizing Labeling Cost in Self-Admitted Technical Debt Identification using Semi-Supervised Learning](https://mlsanity.com/paper/2201.10592/) (2022)
- [Correction of "A Comparative Study to Benchmark Cross-project Defect Prediction Approaches"](https://mlsanity.com/paper/1707.09281/) (2017)
- [Better Software Analytics via "DUO": Data Mining Algorithms Using/Used-by Optimizers](https://mlsanity.com/paper/1812.01550/) (2018)
- [Early Life Cycle Software Defect Prediction. Why? How?](https://mlsanity.com/paper/2011.13071/) (2020)
- [Mining Scientific Workflows for Anomalous Data Transfers](https://mlsanity.com/paper/2103.12221/) (2021)
- [Why is Differential Evolution Better than Grid Search for Tuning Defect Predictors?](https://mlsanity.com/paper/1609.02613/) (2016)
- [Methods for Stabilizing Models across Large Samples of Projects (with case studies on Predicting Defect and Project Health)](https://mlsanity.com/paper/1911.04250/) (2019)
- [Predicting Health Indicators for Open Source Projects (using Hyperparameter Optimization)](https://mlsanity.com/paper/2006.07240/) (2020)
- [Development Effort Estimation in Free/Open Source Software from Activity in Version Control Systems](https://mlsanity.com/paper/2203.09898/) (2022)

## Built on by (most cited first, 3 of 3)

- [Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective](https://mlsanity.com/paper/2412.16888/) (2024)
- [Evaluating Useful Surrogate Models for Configuration Tuning Beyond Accuracy: A Fitness Landscape Analysis Perspective](https://mlsanity.com/paper/2509.21945/) (2025)
- [When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets](https://mlsanity.com/paper/2607.16476/) (2026)

## Abstract (arXiv)

When data is scarce, software analytics can make many mistakes. For example, consider learning predictors for open source project health (e.g. the number of closed pull requests in twelve months time). The training data for this task may be very small (e.g. five years of data, collected every month means just 60 rows of training data). The models generated from such tiny data sets can make many prediction errors. Those errors can be tamed by a em landscape analysis that selects better learner control parameters. Our niSNEAK tool (a)~clusters the data to find the general landscape of the hyperparameters; then (b)~explores a few representatives from each part of that landscape. niSNEAK is both faster and more effective than prior state-of-the-art hyperparameter optimization algorithms (e.g. FLASH, HYPEROPT, OPTUNA). The configurations found by niSNEAK have far less error than other methods. For example, for project health indicators such as C= number of commits; I=number of closed issues, and R=number of closed pull requests, niSNEAK's 12 month prediction errors are I=0%, R=33%,C=47% Based on the above, we recommend landscape analytics (e.g. niSNEAK) especially when learning from very small data sets. This paper only explores the application of niSNEAK to project health. That said, we see nothing in principle that prevents the application of this technique to a wider range of problems. To assist other researchers in repeating, improving, or even refuting our results, all our scripts and data are available on GitHub at https://github.com/zxcv123456qwe/niSneak