# A Framework for Generating Informative Benchmark Instances

arXiv: https://arxiv.org/abs/2205.14753 · HTML: https://mlsanity.com/paper/2205.14753/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 48 references that are in the map.
- First posted on arXiv 29 May 2022.

## Builds on (most cited first, 5 of 5)

- [Benchmarking in Optimization: Best Practice and Open Issues](https://mlsanity.com/paper/2007.03488/) (2020)
- [Best practices for comparing optimization algorithms](https://mlsanity.com/paper/1709.08242/) (2017)
- [An Enhanced Features Extractor for a Portfolio of Constraint Solvers](https://mlsanity.com/paper/1308.0227/) (2013)
- [sunny-as2: Enhancing SUNNY for Algorithm Selection](https://mlsanity.com/paper/2009.03107/) (2020)
- [Generating Instances with Performance Differences for More Than Just Two Algorithms](https://mlsanity.com/paper/2104.14275/) (2021)

## Built on by (most cited first, 4 of 4)

- [Generating Streamlining Constraints with Large Language Models](https://mlsanity.com/paper/2408.10268/) (2024)
- [Frugal Algorithm Selection](https://mlsanity.com/paper/2405.11059/) (2024)
- [Automatic Feature Learning for Essence: a Case Study on Car Sequencing](https://mlsanity.com/paper/2409.15158/) (2024)
- [Streamlined Constraint Reasoning via CNN Pattern Recognition on Enumerated Solutions](https://mlsanity.com/paper/2605.19895/) (2026)

## Abstract (arXiv)

Benchmarking is an important tool for assessing the relative performance of alternative solving approaches. However, the utility of benchmarking is limited by the quantity and quality of the available problem instances. Modern constraint programming languages typically allow the specification of a class-level model that is parameterised over instance data. This separation presents an opportunity for automated approaches to generate instance data that define instances that are graded (solvable at a certain difficulty level for a solver) or can discriminate between two solving approaches. In this paper, we introduce a framework that combines these two properties to generate a large number of benchmark instances, purposely generated for effective and informative benchmarking. We use five problems that were used in the MiniZinc competition to demonstrate the usage of our framework. In addition to producing a ranking among solvers, our framework gives a broader understanding of the behaviour of each solver for the whole instance space; for example by finding subsets of instances where the solver performance significantly varies from its average performance.