Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Similar to other machine learning frameworks, Offline Reinforcement Learning (RL) is shown to be vulnerable to poisoning attacks, due to its reliance on externally sourced datasets, a vulnerability that is exacerbated by its sequential nature. To mitigate the risks posed by RL poisoning, we extend certified defenses to provide larger guarantees against adversarial manipulation, ensuring robustness for both per-state actions, and the overall expected cumulative reward. Our approach leverages properties of Differential Privacy, in a manner that allows this work to span both continuous and discrete spaces, as well as stochastic and deterministic environments -- significantly expanding the scope and applicability of achievable guarantees. Empirical evaluations demonstrate that our approach ensures the performance drops to no more than 50% with up to 7% of the training data poisoned, significantly improving over the 0.008% in prior work~citepwucopa2022, while producing certified radii that is 5 times larger as well. This highlights the potential of our framework to enhance safety and reliability in offline RL.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 60 references that are in the map. First posted on arXiv 27 May 2025.
Builds on
43 of 60 references.
Closest ancestors · 9
Same direction, newest first
- 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximation23 Oct
- 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
- 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets7 Oct
- 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning4 Jun
- 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks16 Mar
- 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Corruption-Robust Offline Reinforcement Learning11 Jun
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 34
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2016Deep Learning with Differential Privacycited by 2,682
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
- 2017Renyi Differential Privacycited by 643
- 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
- 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
- 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
- 2019Reinforcement Learning in Healthcare: A Surveycited by 308
- 2018Learning to Drive in a Daycited by 266
- 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
22 more
- 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
- 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
- 2021Numerical Composition of Differential Privacycited by 174
- 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
- 2020Deep Partition Aggregation: Provable Defense against General Poisoning Attackscited by 112
- 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
- 2020Intrinsic Certified Robustness of Bagging against Data Poisoning Attackscited by 90
- 2019Hypothesis Testing Interpretations and Renyi Differential Privacycited by 84
- 2019Data Poisoning against Differentially-Private Learners: Attacks and Defensescited by 75
- 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
- 2020Private Reinforcement Learning with PAC and Regret Guaranteescited by 51
- 2020Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systemscited by 35
- 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
- 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
- 2020Local Differential Privacy for Regret Minimization in Reinforcement Learningcited by 22
- 2021Defense Against Reward Poisoning Attacks in Reinforcement Learningcited by 20
- 2016Differentially Private Policy Evaluationcited by 19
- 2022Double Bubble, Toil and Trouble: Enhancing Certified Robustness through Transitivitycited by 15
- 2020Locally Private Distributed Reinforcement Learningcited by 9
- 2023Enhancing the Antidote: Improved Pointwise Certifications against Poisoning Attackscited by 6
- 2023Et Tu Certifications: Robustness Certificates Yield Better Adversarial Examplescited by 4
- 2023It's Simplex! Disaggregating Measures to Improve Certified Robustnesscited by 3
Built on by
Citing papers per year: 2025: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning
- 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
- 2022Offline Reinforcement Learning with Differential Privacy
- 2023Robust Offline Reinforcement Learning -- Certify the Confidence Interval
- 2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learning
- 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
- 2026Sequential Data Poisoning in LLM Post-Training
- 2024Preserving Expert-Level Privacy in Offline Reinforcement Learning