# System Safety Monitoring of Learned Components Using Temporal Metric Forecasting

arXiv: https://arxiv.org/abs/2405.13254 · HTML: https://mlsanity.com/paper/2405.13254/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 104 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [A Multi-Horizon Quantile Recurrent Forecaster](https://mlsanity.com/paper/1711.11053/) (2017)
- [Deep Learning for Time Series Forecasting: Tutorial and Literature Survey](https://mlsanity.com/paper/2004.10240/) (2020)
- [Anomaly Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2204.07974/) (2022)
- [Deep Neural Network Compression for Aircraft Collision Avoidance Systems](https://mlsanity.com/paper/1810.04240/) (2018)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [Criteria for Classifying Forecasting Methods](https://mlsanity.com/paper/2212.03523/) (2022)
- [Fully neuromorphic vision and control for autonomous drone flight](https://mlsanity.com/paper/2303.08778/) (2023)
- [Deep Learning-Based Anomaly Detection in Cyber-Physical Systems: Progress and Opportunities](https://mlsanity.com/paper/2003.13213/) (2020)
- [Verification of Image-based Neural Network Controllers Using Generative Models](https://mlsanity.com/paper/2105.07091/) (2021)
- [Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends](https://mlsanity.com/paper/2101.01364/) (2021)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Resource-Efficient Neural Networks for Embedded Systems](https://mlsanity.com/paper/2001.03048/) (2020)
- [Towards Structured Evaluation of Deep Neural Network Supervisors](https://mlsanity.com/paper/1903.01263/) (2019)
- [Self-Checking Deep Neural Networks in Deployment](https://mlsanity.com/paper/2103.02371/) (2021)
- [Closed-loop Analysis of Vision-based Autonomous Systems: A Case Study](https://mlsanity.com/paper/2302.04634/) (2023)
- [DeepGuard: A Framework for Safeguarding Autonomous Driving Systems from Inconsistent Behavior](https://mlsanity.com/paper/2111.09533/) (2021)
- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)

## Built on by (most cited first, 4 of 4)

- [Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies](https://mlsanity.com/paper/2603.16952/) (2026)
- [Digital Twin-based Out-of-Distribution Detection in Autonomous Vessels](https://mlsanity.com/paper/2504.19816/) (2025)
- [Label Forensics: Interpreting Hard Labels in Black-Box Text Classifier](https://mlsanity.com/paper/2512.01514/) (2025)
- [Uncertainty-Guided Label Rebalancing for CPS Safety Monitoring](https://mlsanity.com/paper/2603.25670/) (2026)

## Abstract (arXiv)

In learning-enabled autonomous systems, safety monitoring of learned components is crucial to ensure their outputs do not lead to system safety violations, given the operational context of the system. However, developing a safety monitor for practical deployment in real-world applications is challenging. This is due to limited access to internal workings and training data of the learned component. Furthermore, safety monitors should predict safety violations with low latency, while consuming a reasonable amount of computation. To address the challenges, we propose a safety monitoring method based on probabilistic time series forecasting. Given the learned component outputs and an operational context, we empirically investigate different Deep Learning (DL)-based probabilistic forecasting to predict the objective measure capturing the satisfaction or violation of a safety requirement (safety metric). We empirically evaluate safety metric and violation prediction accuracy, and inference latency and resource usage of four state-of-the-art models, with varying horizons, using autonomous aviation and autonomous driving case studies. Our results suggest that probabilistic forecasting of safety metrics, given learned component outputs and scenarios, is effective for safety monitoring. Furthermore, for both case studies, Temporal Fusion Transformer (TFT) was the most accurate model for predicting imminent safety violations, with acceptable latency and resource consumption.