# SafeML: Safety Monitoring of Machine Learning Classifiers through Statistical Difference Measure

arXiv: https://arxiv.org/abs/2005.13166 · HTML: https://mlsanity.com/paper/2005.13166/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 16 of 16)

- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2308.02594/) (2023)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward](https://mlsanity.com/paper/2404.08517/) (2024)
- [Keep your Distance: Determining Sampling and Distance Thresholds in Machine Learning Monitoring](https://mlsanity.com/paper/2207.05078/) (2022)
- [StaDRe and StaDRo: Reliability and Robustness Estimation of ML-based Forecasting using Statistical Distance Measures](https://mlsanity.com/paper/2206.11116/) (2022)
- [Scope Compliance Uncertainty Estimate](https://mlsanity.com/paper/2312.10801/) (2023)
- [Out-of-Distribution Detection for Safety Assurance of AI and Autonomous Systems](https://mlsanity.com/paper/2510.21254/) (2025)
- [Integrating Testing and Operation-related Quantitative Evidences in Assurance Cases to Argue Safety of Data-Driven AI/ML Components](https://mlsanity.com/paper/2202.05313/) (2022)
- [A Deep Learning Framework for Wind Turbine Repair Action Prediction Using Alarm Sequences and Long Short Term Memory Algorithms](https://mlsanity.com/paper/2207.09457/) (2022)
- [Q-SafeML: Safety Assessment of Quantum Machine Learning via Quantum Distance Metrics](https://mlsanity.com/paper/2509.04536/) (2025)
- [Online Dynamic Reliability Evaluation of Wind Turbines based on Drone-assisted Monitoring](https://mlsanity.com/paper/2211.13258/) (2022)
- [Ensembling Uncertainty Measures to Improve Safety of Black-Box Classifiers](https://mlsanity.com/paper/2308.12065/) (2023)
- [Towards Robust Training Datasets for Machine Learning with Ontologies: A Case Study for Emergency Road Vehicle Detection](https://mlsanity.com/paper/2406.15268/) (2024)
- [Incorporating Failure of Machine Learning in Dynamic Probabilistic Safety Assurance](https://mlsanity.com/paper/2506.06868/) (2025)
- [Safer Skin Lesion Classification with Global Class Activation Probability Map Evaluation and SafeML](https://mlsanity.com/paper/2508.20776/) (2025)

## Abstract (arXiv)

Ensuring safety and explainability of machine learning (ML) is a topic of increasing relevance as data-driven applications venture into safety-critical application domains, traditionally committed to high safety standards that are not satisfied with an exclusive testing approach of otherwise inaccessible black-box systems. Especially the interaction between safety and security is a central challenge, as security violations can lead to compromised safety. The contribution of this paper to addressing both safety and security within a single concept of protection applicable during the operation of ML systems is active monitoring of the behaviour and the operational context of the data-driven system based on distance measures of the Empirical Cumulative Distribution Function (ECDF). We investigate abstract datasets (XOR, Spiral, Circle) and current security-specific datasets for intrusion detection (CICIDS2017) of simulated network traffic, using distributional shift detection measures including the Kolmogorov-Smirnov, Kuiper, Anderson-Darling, Wasserstein and mixed Wasserstein-Anderson-Darling measures. Our preliminary findings indicate that the approach can provide a basis for detecting whether the application context of an ML component is valid in the safety-security. Our preliminary code and results are available at https://github.com/ISorokos/SafeML.