mlsanityData to 7 Oct 2026

ML4ML: Automated Invariance Testing for Machine Learning Models

arXiv 2109.12926 ↗First posted 27 Sep 2021cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

In machine learning (ML) workflows, determining the invariance qualities of an ML model is a common testing procedure. Traditionally, invariance qualities are evaluated using simple formula-based scores, e.g., accuracy. In this paper, we show that testing the invariance qualities of ML models may result in complex visual patterns that cannot be classified using simple formulas. In order to test ML models by analyzing such visual patterns automatically using other ML models, we propose a systematic framework that is applicable to a variety of invariance qualities. We demonstrate the effectiveness and feasibility of the framework by developing ML4ML models (assessors) for determining rotation-, brightness-, and size-variances of a collection of neural networks. Our testing results show that the trained ML4ML assessors can perform such analytical tasks with sufficient accuracy.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 65 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

25 of 65 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  7. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  8. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  9. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  10. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  11. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  12. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
13 more
  1. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 123
  2. 2020Evading Deepfake-Image Detectors with White- and Black-Box Attackscited by 80
  3. 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 67
  4. 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 66
  5. 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 64
  6. 2018Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testingcited by 30
  7. 2018Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolutioncited by 16
  8. 2019Detecting Overfitting via Adversarial Examplescited by 15
  9. 2019Scale-wise Convolution for Image Restorationcited by 14
  10. 2018Transfer Learning for Action Unit Recognitioncited by 2
  11. 2020Do CNNs Encode Data Augmentations?cited by 2
  12. 2020Rotation-Invariant Gait Identification with Quaternion Convolutional Neural Networkscited by 2
  13. 2018Simultaneous Adversarial Training - Learn from Others Mistakescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

2 papers.

  1. 2026Discriminative Representation Learning for Clinical Predictioncited by 2
  2. 2022Background Invariance Testing According to Semantic Proximitycited by 0