# Thinking Beyond Distributions in Testing Machine Learned Models

arXiv: https://arxiv.org/abs/2112.03057 · HTML: https://mlsanity.com/paper/2112.03057/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 20 of 20)

- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [Perturbation Sensitivity Analysis to Detect Unintended Model Biases](https://mlsanity.com/paper/1910.04210/) (2019)
- [Understanding Failures of Deep Networks via Robust Feature Extraction](https://mlsanity.com/paper/2012.01750/) (2020)
- [Extending the Machine Learning Abstraction Boundary: A Complex Systems Approach to Incorporate Societal Context](https://mlsanity.com/paper/2006.09663/) (2020)
- [Representativeness in Statistics, Politics, and Machine Learning](https://mlsanity.com/paper/2101.03827/) (2021)
- [Adversarial Framing for Image and Video Classification](https://mlsanity.com/paper/1812.04599/) (2018)
- [Meltdown](https://mlsanity.com/paper/1801.01207/) (2018)

## Built on by (most cited first, 5 of 5)

- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML](https://mlsanity.com/paper/2210.03535/) (2022)
- [Deep Learning Safety Concerns in Automated Driving Perception](https://mlsanity.com/paper/2309.03774/) (2023)
- [Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models](https://mlsanity.com/paper/2410.24005/) (2024)
- [Measuring and signing fairness as performance under multiple stakeholder distributions](https://mlsanity.com/paper/2207.09960/) (2022)

## Abstract (arXiv)

Testing practices within the machine learning (ML) community have centered around assessing a learned model's predictive performance measured against a test dataset, often drawn from the same distribution as the training dataset. While recent work on robustness and fairness testing within the ML community has pointed to the importance of testing against distributional shifts, these efforts also focus on estimating the likelihood of the model making an error against a reference dataset/distribution. We argue that this view of testing actively discourages researchers and developers from looking into other sources of robustness failures, for instance corner cases which may have severe undesirable impacts. We draw parallels with decades of work within software engineering testing focused on assessing a software system against various stress conditions, including corner cases, as opposed to solely focusing on average-case behaviour. Finally, we put forth a set of recommendations to broaden the view of machine learning testing to a rigorous practice.