# Learning Run-time Safety Monitors for Machine Learning Components

arXiv: https://arxiv.org/abs/2406.16220 · HTML: https://mlsanity.com/paper/2406.16220/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Data Augmentation Using GANs](https://mlsanity.com/paper/1904.09135/) (2019)
- [Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)](https://mlsanity.com/paper/2102.01564/) (2021)
- [Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Tests](https://mlsanity.com/paper/2107.06929/) (2021)
- [Out-Of-Distribution Detection Is Not All You Need](https://mlsanity.com/paper/2211.16158/) (2022)
- [Optimal Ratio for Data Splitting](https://mlsanity.com/paper/2202.03326/) (2022)
- [A Less Biased Evaluation of Out-of-distribution Sample Detectors](https://mlsanity.com/paper/1809.04729/) (2018)
- [DeepCert: Verification of Contextually Relevant Robustness for Neural Network Image Classifiers](https://mlsanity.com/paper/2103.01629/) (2021)
- [Combining GANs and AutoEncoders for Efficient Anomaly Detection](https://mlsanity.com/paper/2011.08102/) (2020)
- [Runtime Monitoring for Out-of-Distribution Detection in Object Detection Neural Networks](https://mlsanity.com/paper/2212.07773/) (2022)

## Abstract (arXiv)

For machine learning components used as part of autonomous systems (AS) in carrying out critical tasks it is crucial that assurance of the models can be maintained in the face of post-deployment changes (such as changes in the operating environment of the system). A critical part of this is to be able to monitor when the performance of the model at runtime (as a result of changes) poses a safety risk to the system. This is a particularly difficult challenge when ground truth is unavailable at runtime. In this paper we introduce a process for creating safety monitors for ML components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output. We demonstrate the viability of our approach through some initial experiments using publicly available speed sign datasets.