mlsanityData to 7 Oct 2026

COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks

arXiv 2203.08398 ↗First posted 16 Mar 2022cs.LG cs.CRcited by 25

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

As reinforcement learning (RL) has achieved near human-level performance in a variety of tasks, its robustness has raised great attention. While a vast body of research has explored test-time (evasion) attacks in RL and corresponding defenses, its robustness against training-time (poisoning) attacks remains largely unanswered. In this work, we focus on certifying the robustness of offline RL in the presence of poisoning attacks, where a subset of training trajectories could be arbitrarily manipulated. We propose the first certification framework, COPA, to certify the number of poisoning trajectories that can be tolerated regarding different certification criteria. Given the complex structure of RL, we propose two certification criteria: per-state action stability and cumulative reward bound. To further improve the certification, we propose new partition and aggregation protocols to train robust policies. We further prove that some of the proposed certification methods are theoretically tight and some are NP-Complete problems. We leverage COPA to certify three RL environments trained with different algorithms and conclude: (1) The proposed robust aggregation protocols such as temporal aggregation can significantly improve the certifications; (2) Our certification for both per-state action stability and cumulative reward bound are efficient and tight; (3) The certification for different training algorithms and environments are different, implying their intrinsic robustness properties. All experimental results are available at https://copa-leaderboard.github.io.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 62 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

43 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2019Policy Poisoning in Batch Reinforcement Learning and Control13 Oct

Other references · 40

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2019Solving Rubik's Cube with a Robot Handcited by 699
  8. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  11. 2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 322
  12. 2017Adversarial Attacks on Neural Network Policiescited by 314
28 more
  1. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  2. 2017Certified Defenses for Data Poisoning Attackscited by 288
  3. 2017Noisy Networks for Explorationcited by 262
  4. 2018Towards Fast Computation of Certified Robustness for ReLU Networkscited by 254
  5. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  6. 2021Mastering Atari Games with Limited Datacited by 235
  7. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  8. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  9. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
  10. 2020Adversarial Machine Learning -- Industry Perspectivescited by 121
  11. 2020RAB: Provable Robustness Against Backdoor Attackscited by 115
  12. 2020Deep Partition Aggregation: Provable Defense against General Poisoning Attackscited by 112
  13. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  14. 2020Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attackscited by 109
  15. 2017Delving into adversarial attacks on deep policiescited by 104
  16. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  17. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  18. 2020Robust Deep Reinforcement Learning through Adversarial Losscited by 73
  19. 2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 66
  20. 2019Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signalscited by 50
  21. 2021CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothingcited by 41
  22. 2020Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamicscited by 38
  23. 2017Whatever Does Not Kill Deep Reinforcement Learning, Makes It Strongercited by 35
  24. 2020Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systemscited by 35
  25. 2019Online Robustness Training for Deep Reinforcement Learningcited by 26
  26. 2020Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learningcited by 23
  27. 2021Defense Against Reward Poisoning Attacks in Reinforcement Learningcited by 20
  28. 2018Mitigation of Policy Manipulation Attacks on Deep Q-Networks with Parameter-Space Noisecited by 7

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 8; 2024: 7; 2025: 6.

25 papers.

Sort:

By area:

  1. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  2. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  3. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  4. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  5. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  6. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  7. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  8. 2023Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influencecited by 16
  9. 2023Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attackscited by 14
  10. 2024COLEP: Certifiably Robust Learning-Reasoning Conformal Prediction via Probabilistic Circuitscited by 9
  11. 2024Beyond the Edge: An Advanced Exploration of Reinforcement Learning for Mobile Edge Computing, its Applications, and Future Research Trajectoriescited by 9
  12. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
13 more
  1. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  2. 2025On Corruption-Robustness in Performative Reinforcement Learningcited by 6
  3. 2024Data Poisoning Attacks on Off-Policy Policy Evaluation Methodscited by 4
  4. 2024Adversarially Robust Decision Transformercited by 4
  5. 2025Position: Certified Robustness Does Not (Yet) Imply Model Securitycited by 4
  6. 2023Robust Multi-agent Communication via Multi-view Message Certificationcited by 3
  7. 2025Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learningcited by 3
  8. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2
  9. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  10. 2022Chaos Theory and Adversarial Robustnesscited by 1
  11. 2023Policy Resilience to Environment Poisoning Attacks on Reinforcement Learningcited by 1
  12. 2023Robust Offline Reinforcement Learning -- Certify the Confidence Intervalcited by 0
  13. 2025Optimal Perturbation Budget Allocation for Data Poisoning in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning
  2. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  3. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  4. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
  5. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  6. 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy
  7. 2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learning
  8. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses