# Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretation

arXiv: https://arxiv.org/abs/2301.11374 · HTML: https://mlsanity.com/paper/2301.11374/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 30 of 33)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Efficient Neural Network Robustness Certification with General Activation Functions](https://mlsanity.com/paper/1811.00866/) (2018)
- [On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models](https://mlsanity.com/paper/1810.12715/) (2018)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Tactics of Adversarial Attack on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1703.06748/) (2017)
- [Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methods](https://mlsanity.com/paper/2202.11762/) (2022)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions](https://mlsanity.com/paper/2109.06697/) (2021)
- [Robust Reinforcement Learning for Continuous Control with Model Misspecification](https://mlsanity.com/paper/1906.07516/) (2019)
- [Robust Deep Reinforcement Learning through Adversarial Loss](https://mlsanity.com/paper/2008.01976/) (2020)
- [Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL](https://mlsanity.com/paper/2106.05087/) (2021)
- [Certified Adversarial Robustness for Deep Reinforcement Learning](https://mlsanity.com/paper/1910.12908/) (2019)
- [MBRL-Lib: A Modular Library for Model-based Reinforcement Learning](https://mlsanity.com/paper/2104.10159/) (2021)
- [CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing](https://mlsanity.com/paper/2106.09292/) (2021)
- [Policy Smoothing for Provably Robust Reinforcement Learning](https://mlsanity.com/paper/2106.11420/) (2021)
- [Control Regularization for Reduced Variance Reinforcement Learning](https://mlsanity.com/paper/1905.05380/) (2019)
- [Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics](https://mlsanity.com/paper/2107.04989/) (2021)
- [Online Robustness Training for Deep Reinforcement Learning](https://mlsanity.com/paper/1911.00887/) (2019)
- [Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/2004.06496/) (2020)
- [Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions](https://mlsanity.com/paper/1903.09885/) (2019)

## Built on by (most cited first, 3 of 3)

- [Canopy: Property-Driven Learning for Congestion Control](https://mlsanity.com/paper/2412.10915/) (2024)
- [Training Verifiably Robust Agents Using Set-Based Reinforcement Learning](https://mlsanity.com/paper/2408.09112/) (2024)
- [FoSeRL: Formal Sequential Robustness Certification for Reinforcement Learning Policies](https://mlsanity.com/paper/2610.04754/) (2026)

## Abstract (arXiv)

We present a reinforcement learning (RL) framework in which the learned policy comes with a machine-checkable certificate of provable adversarial robustness. Our approach, called CAROL, learns a model of the environment. In each learning iteration, it uses the current version of this model and an external abstract interpreter to construct a differentiable signal for provable robustness. This signal is used to guide learning, and the abstract interpretation used to construct it directly leads to the robustness certificate returned at convergence. We give a theoretical analysis that bounds the worst-case accumulative reward of CAROL. We also experimentally evaluate CAROL on four MuJoCo environments with continuous state and action spaces. On these tasks, CAROL learns policies that, when contrasted with policies from the state-of-the-art robust RL algorithms, exhibit: (i) markedly enhanced certified performance lower bounds; and (ii) comparable performance under empirical adversarial attacks.