mlsanityData to 7 Oct 2026

Distribution Awareness for AI System Testing

arXiv 2105.02540 ↗First posted 6 May 2021cs.SE cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

As Deep Learning (DL) is continuously adopted in many safety critical applications, its quality and reliability start to raise concerns. Similar to the traditional software development process, testing the DL software to uncover its defects at an early stage is an effective way to reduce risks after deployment. Although recent progress has been made in designing novel testing techniques for DL software, the distribution of generated test data is not taken into consideration. It is therefore hard to judge whether the identified errors are indeed meaningful errors to the DL application. Therefore, we propose a new OOD-guided testing technique which aims to generate new unseen test cases relevant to the underlying DL system task. Our results show that this technique is able to filter up to 55.44% of error test case on CIFAR-10 and is 10.05% more effective in enhancing robustness.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

19 of 30 references.

References · 19

  1. 2016Densely Connected Convolutional Networkscited by 6,691
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  6. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  7. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  8. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  9. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  10. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  11. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  12. 2019Input complexity and out-of-distribution detection with likelihood-based generative modelscited by 186
7 more
  1. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
  2. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 123
  3. 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 67
  4. 2018Concolic Testing for Deep Neural Networkscited by 51
  5. 2018WAIC, but Why? Generative Ensembles for Robust Anomaly Detectioncited by 25
  6. 2019Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similaritycited by 12
  7. 2019Analysis of Confident-Classifiers for Out-of-distribution Detectioncited by 7

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 1; 2025: 1.

6 papers.

  1. 2021Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurancecited by 16
  2. 2022Hierarchical Distribution-Aware Testing of Deep Learningcited by 8
  3. 2023What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systemscited by 6
  4. 2025Hierarchical Testing with Rabbit Optimization for Industrial Cyber-Physical Systemscited by 4
  5. 2024Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCRcited by 2
  6. 2023Distribution-aware Fairness Test Generationcited by 0