# Correct-by-Construction Runtime Enforcement in AI -- A Survey

arXiv: https://arxiv.org/abs/2208.14426 · HTML: https://mlsanity.com/paper/2208.14426/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 71 references that are in the map.
- First posted on arXiv 30 Aug 2022.

## Builds on (most cited first, 30 of 43)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Grad-CAM: Why did you say that?](https://mlsanity.com/paper/1611.07450/) (2016)
- [Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers](https://mlsanity.com/paper/1809.03576/) (2018)
- [Learning in Implicit Generative Models](https://mlsanity.com/paper/1610.03483/) (2016)
- [Safe Multi-Agent Reinforcement Learning via Shielding](https://mlsanity.com/paper/2101.11196/) (2021)
- [Cautious Reinforcement Learning with Logical Constraints](https://mlsanity.com/paper/2002.12156/) (2020)
- [Shield Synthesis: Runtime Enforcement for Reactive Systems](https://mlsanity.com/paper/1501.02573/) (2015)
- [Out-of-Distribution Detection for Automotive Perception](https://mlsanity.com/paper/2011.01413/) (2020)
- [Towards Verified Artificial Intelligence](https://mlsanity.com/paper/1606.08514/) (2016)
- [Certified Reinforcement Learning with Logic Guidance](https://mlsanity.com/paper/1902.00778/) (2019)
- [Outside the Box: Abstraction-Based Monitoring of Neural Networks](https://mlsanity.com/paper/1911.09032/) (2019)

## Built on by (most cited first, 12 of 12)

- [Compositional Shielding and Reinforcement Learning for Multi-Agent Systems](https://mlsanity.com/paper/2410.10460/) (2024)
- [Provably Safe Neural Network Controllers via Differential Dynamic Logic](https://mlsanity.com/paper/2402.10998/) (2024)
- [CommonPower: A Framework for Safe Data-Driven Smart Grid Control](https://mlsanity.com/paper/2406.03231/) (2024)
- [Safety Shielding under Delayed Observation](https://mlsanity.com/paper/2307.02164/) (2023)
- [Composing Reinforcement Learning Policies, with Formal Guarantees](https://mlsanity.com/paper/2402.13785/) (2024)
- [Verification of Autonomous Neural Car Control with KeYmaera X](https://mlsanity.com/paper/2504.03272/) (2025)
- [Follow the STARs: Dynamic ω-Regular Shielding of Learned Policies](https://mlsanity.com/paper/2505.14689/) (2025)
- [Experiments with Detecting and Mitigating AI Deception](https://mlsanity.com/paper/2306.14816/) (2023)
- [Explainably Safe Reinforcement Learning](https://mlsanity.com/paper/2606.04634/) (2026)
- [Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents](https://mlsanity.com/paper/2609.39717/) (2026)
- [MENTOR: Procedural Shielding of Tool-Using LLM Agents](https://mlsanity.com/paper/2610.17039/) (2026)
- [One Monitor Does Not Fit All: Formalism-Agnostic Runtime Verification for Agentic AI](https://mlsanity.com/paper/2610.19818/) (2026)

## Abstract (arXiv)

Runtime enforcement refers to the theories, techniques, and tools for enforcing correct behavior with respect to a formal specification of systems at runtime. In this paper, we are interested in techniques for constructing runtime enforcers for the concrete application domain of enforcing safety in AI. We discuss how safety is traditionally handled in the field of AI and how more formal guarantees on the safety of a self-learning agent can be given by integrating a runtime enforcer. We survey a selection of work on such enforcers, where we distinguish between approaches for discrete and continuous action spaces. The purpose of this paper is to foster a better understanding of advantages and limitations of different enforcement techniques, focusing on the specific challenges that arise due to their application in AI. Finally, we present some open challenges and avenues for future work.