# QuantifyML: How Good is my Machine Learning Model?

arXiv: https://arxiv.org/abs/2110.12588 · HTML: https://mlsanity.com/paper/2110.12588/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 8 of 8)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [A Scalable Approximate Model Counter](https://mlsanity.com/paper/1306.5726/) (2013)
- [Toward Adversarial Robustness by Diversity in an Ensemble of Specialized Deep Neural Networks](https://mlsanity.com/paper/2005.08321/) (2020)
- [Projected Model Counting](https://mlsanity.com/paper/1507.07648/) (2015)
- [A Study of the Learnability of Relational Properties: Model Counting Meets Machine Learning (MCML)](https://mlsanity.com/paper/1912.11580/) (2019)

## Abstract (arXiv)

The efficacy of machine learning models is typically determined by computing their accuracy on test data sets. However, this may often be misleading, since the test data may not be representative of the problem that is being studied. With QuantifyML we aim to precisely quantify the extent to which machine learning models have learned and generalized from the given data. Given a trained model, QuantifyML translates it into a C program and feeds it to the CBMC model checker to produce a formula in Conjunctive Normal Form (CNF). The formula is analyzed with off-the-shelf model counters to obtain precise counts with respect to different model behavior. QuantifyML enables i) evaluating learnability by comparing the counts for the outputs to ground truth, expressed as logical predicates, ii) comparing the performance of models built with different machine learning algorithms (decision-trees vs. neural networks), and iii) quantifying the safety and robustness of models.