# Keep your Distance: Determining Sampling and Distance Thresholds in Machine Learning Monitoring

arXiv: https://arxiv.org/abs/2207.05078 · HTML: https://mlsanity.com/paper/2207.05078/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 11 Jul 2022.

## Builds on (most cited first, 13 of 13)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [On Wasserstein Two Sample Testing and Related Families of Nonparametric Tests](https://mlsanity.com/paper/1509.02237/) (2015)
- [Feature-Guided Black-Box Safety Testing of Deep Neural Networks](https://mlsanity.com/paper/1710.07859/) (2017)
- [Importance-Driven Deep Learning System Testing](https://mlsanity.com/paper/2002.03433/) (2020)
- [SafeML: Safety Monitoring of Machine Learning Classifiers through Statistical Difference Measure](https://mlsanity.com/paper/2005.13166/) (2020)
- [Efficient Certification of Spatial Robustness](https://mlsanity.com/paper/2009.09318/) (2020)
- [Assessing the Reliability of Deep Learning Classifiers Through Robustness Evaluation and Operational Profiles](https://mlsanity.com/paper/2106.01258/) (2021)
- [Detecting Operational Adversarial Examples for Reliable Deep Learning](https://mlsanity.com/paper/2104.06015/) (2021)
- [A Provable Defense for Deep Residual Networks](https://mlsanity.com/paper/1903.12519/) (2019)
- [Detection and Mitigation of Rare Subclasses in Deep Neural Network Classifiers](https://mlsanity.com/paper/1911.12780/) (2019)
- [Autoencoder-based Semantic Novelty Detection: Towards Dependable AI-based Systems](https://mlsanity.com/paper/2108.10851/) (2021)

## Built on by (most cited first, 4 of 4)

- [SafeLLM: Domain-Specific Safety Monitoring for Large Language Models: A Case Study of Offshore Wind Maintenance](https://mlsanity.com/paper/2410.10852/) (2024)
- [Scope Compliance Uncertainty Estimate](https://mlsanity.com/paper/2312.10801/) (2023)
- [Q-SafeML: Safety Assessment of Quantum Machine Learning via Quantum Distance Metrics](https://mlsanity.com/paper/2509.04536/) (2025)
- [Safer Skin Lesion Classification with Global Class Activation Probability Map Evaluation and SafeML](https://mlsanity.com/paper/2508.20776/) (2025)

## Abstract (arXiv)

Machine Learning~(ML) has provided promising results in recent years across different applications and domains. However, in many cases, qualities such as reliability or even safety need to be ensured. To this end, one important aspect is to determine whether or not ML components are deployed in situations that are appropriate for their application scope. For components whose environments are open and variable, for instance those found in autonomous vehicles, it is therefore important to monitor their operational situation to determine its distance from the ML components' trained scope. If that distance is deemed too great, the application may choose to consider the ML component outcome unreliable and switch to alternatives, e.g. using human operator input instead. SafeML is a model-agnostic approach for performing such monitoring, using distance measures based on statistical testing of the training and operational datasets. Limitations in setting SafeML up properly include the lack of a systematic approach for determining, for a given application, how many operational samples are needed to yield reliable distance information as well as to determine an appropriate distance threshold. In this work, we address these limitations by providing a practical approach and demonstrate its use in a well known traffic sign recognition problem, and on an example using the CARLA open-source automotive simulator.