# Data Synthesis for Testing Black-Box Machine Learning Models

arXiv: https://arxiv.org/abs/2111.02161 · HTML: https://mlsanity.com/paper/2111.02161/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 3 Nov 2021.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 14 of 14)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Fairness Testing: Testing Software for Discrimination](https://mlsanity.com/paper/1709.03221/) (2017)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [Automated Directed Fairness Testing](https://mlsanity.com/paper/1807.00468/) (2018)
- [Enhancing Robustness of Machine Learning Systems via Data Transformations](https://mlsanity.com/paper/1704.02654/) (2017)
- [Fast Online "Next Best Offers" using Deep Learning](https://mlsanity.com/paper/1905.13368/) (2019)

## Built on by (most cited first, 1 of 1)

- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)

## Abstract (arXiv)

The increasing usage of machine learning models raises the question of the reliability of these models. The current practice of testing with limited data is often insufficient. In this paper, we provide a framework for automated test data synthesis to test black-box ML/DL models. We address an important challenge of generating realistic user-controllable data with model agnostic coverage criteria to test a varied set of properties, essentially to increase trust in machine learning models. We experimentally demonstrate the effectiveness of our technique.