# CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing

arXiv: https://arxiv.org/abs/2106.09292 · HTML: https://mlsanity.com/paper/2106.09292/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 90 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 30 of 55)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Certified Robustness to Adversarial Examples with Differential Privacy](https://mlsanity.com/paper/1802.03471/) (2018)
- [Efficient Neural Network Robustness Certification with General Activation Functions](https://mlsanity.com/paper/1811.00866/) (2018)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers](https://mlsanity.com/paper/1906.04584/) (2019)
- [On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models](https://mlsanity.com/paper/1810.12715/) (2018)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Certified Defenses against Adversarial Examples](https://mlsanity.com/paper/1801.09344/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Towards Fast Computation of Certified Robustness for ReLU Networks](https://mlsanity.com/paper/1804.09699/) (2018)
- [Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks](https://mlsanity.com/paper/1705.01320/) (2017)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation](https://mlsanity.com/paper/1705.08475/) (2017)

## Built on by (most cited first, 30 of 42)

- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning](https://mlsanity.com/paper/2210.05927/) (2022)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks](https://mlsanity.com/paper/2203.08398/) (2022)
- [Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum](https://mlsanity.com/paper/2206.10057/) (2022)
- [Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attacks](https://mlsanity.com/paper/2302.13851/) (2023)
- [Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attacks](https://mlsanity.com/paper/2207.10170/) (2022)
- [Optimal Attack and Defense for Reinforcement Learning](https://mlsanity.com/paper/2312.00198/) (2023)
- [Confidence-aware Training of Smoothed Classifiers for Certified Robustness](https://mlsanity.com/paper/2212.09000/) (2022)
- [COLEP: Certifiably Robust Learning-Reasoning Conformal Prediction via Probabilistic Circuits](https://mlsanity.com/paper/2403.11348/) (2024)
- [Certifiably Robust Policy Learning against Adversarial Communication in Multi-agent Systems](https://mlsanity.com/paper/2206.10158/) (2022)
- [Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.11746/) (2022)
- [Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL](https://mlsanity.com/paper/2305.17342/) (2023)
- [Reward Certification for Policy Smoothed Reinforcement Learning](https://mlsanity.com/paper/2312.06436/) (2023)
- [Improve Robustness of Reinforcement Learning against Observation Perturbations via l_∞ Lipschitz Policy Networks](https://mlsanity.com/paper/2312.08751/) (2023)
- [Certifying Model Accuracy under Distribution Shifts](https://mlsanity.com/paper/2201.12440/) (2022)
- [Corruption-Robust Offline Two-Player Zero-Sum Markov Games](https://mlsanity.com/paper/2403.07933/) (2024)
- [On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks](https://mlsanity.com/paper/2405.11432/) (2024)
- [Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents](https://mlsanity.com/paper/2406.18062/) (2024)
- [FI-ODE: Certifiably Robust Forward Invariance in Neural ODEs](https://mlsanity.com/paper/2210.16940/) (2022)
- [SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2301.04299/) (2023)
- [Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretation](https://mlsanity.com/paper/2301.11374/) (2023)
- [Toward Evaluating Robustness of Reinforcement Learning with Adversarial Policy](https://mlsanity.com/paper/2305.02605/) (2023)
- [Robust Multi-agent Communication via Multi-view Message Certification](https://mlsanity.com/paper/2305.13936/) (2023)
- [Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency](https://mlsanity.com/paper/2608.12939/) (2026)
- [Security of Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2212.06123/) (2022)
- [Mixing Classifiers to Alleviate the Accuracy-Robustness Trade-Off](https://mlsanity.com/paper/2311.15165/) (2023)
- [Robust Deep Reinforcement Learning against Adversarial Behavior Manipulation](https://mlsanity.com/paper/2406.03862/) (2024)
- [An Overview and Prospective Outlook on Robust Training and Certification of Machine Learning Models](https://mlsanity.com/paper/2208.07464/) (2022)
- [Chaos Theory and Adversarial Robustness](https://mlsanity.com/paper/2210.13235/) (2022)

## Abstract (arXiv)

As reinforcement learning (RL) has achieved great success and been even adopted in safety-critical domains such as autonomous vehicles, a range of empirical studies have been conducted to improve its robustness against adversarial attacks. However, how to certify its robustness with theoretical guarantees still remains challenging. In this paper, we present the first unified framework CROP (Certifying Robust Policies for RL) to provide robustness certification on both action and reward levels. In particular, we propose two robustness certification criteria: robustness of per-state actions and lower bound of cumulative rewards. We then develop a local smoothing algorithm for policies derived from Q-functions to guarantee the robustness of actions taken along the trajectory; we also develop a global smoothing algorithm for certifying the lower bound of a finite-horizon cumulative reward, as well as a novel local smoothing algorithm to perform adaptive search in order to obtain tighter reward certification. Empirically, we apply CROP to evaluate several existing empirically robust RL algorithms, including adversarial training and different robust regularization, in four environments (two representative Atari games, Highway, and CartPole). Furthermore, by evaluating these algorithms against adversarial attacks, we demonstrate that our certification are often tight. All experiment results are available at website https://crop-leaderboard.github.io.