mlsanityData to 7 Oct 2026

Data Synthesis for Testing Black-Box Machine Learning Models

arXiv 2111.02161 ↗First posted 3 Nov 2021cs.LG cs.AIcited by 1

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The increasing usage of machine learning models raises the question of the reliability of these models. The current practice of testing with limited data is often insufficient. In this paper, we provide a framework for automated test data synthesis to test black-box ML/DL models. We address an important challenge of generating realistic user-controllable data with model agnostic coverage criteria to test a varied set of properties, essentially to increase trust in machine learning models. We experimentally demonstrate the effectiveness of our technique.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 14 of its 45 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

14 of 45 references.

References · 14

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  4. 2017Counterfactual Fairnesscited by 802
  5. 2014Certifying and removing disparate impactcited by 715
  6. 2019Modeling Tabular data using Conditional GANcited by 695
  7. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 612
  8. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  9. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
  10. 2017Fairness Testing: Testing Software for Discriminationcited by 123
  11. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 123
  12. 2018Automated Directed Fairness Testingcited by 77
2 more
  1. 2017Enhancing Robustness of Machine Learning Systems via Data Transformationscited by 38
  2. 2019Fast Online "Next Best Offers" using Deep Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31

Similar, not cited

Close in wording, no citation either way

  1. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding
  2. 2024A Survey of Data Synthesis Approaches
  3. 2025Program Synthesis via Test-Time Transduction
  4. 2024Towards Automated Machine Learning Research
  5. 2025Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice
  6. 2026Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora
  7. 2022Exploring validation metrics for offline model-based optimisation with diffusion models
  8. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Coding