mlsanityData to 7 Oct 2026

Robust Policy Gradient against Strong Data Corruption

arXiv 2102.05800 ↗First posted 11 Feb 2021cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

We study the problem of robust reinforcement learning under adversarial corruption on both rewards and transitions. Our attack model assumes an textitadaptive adversary who can arbitrarily corrupt the reward and transition at every step within an episode, for at most ε-fraction of the learning episodes. Our attack model is strictly stronger than those considered in prior works. Our first result shows that no algorithm can find a better than O(ε)-optimal policy under our attack model. Next, we show that surprisingly the natural policy gradient (NPG) method retains a natural robustness property if the reward corruption is bounded, and can find an O(√ε)-optimal policy. Consequently, we develop a Filtered Policy Gradient (FPG) algorithm that can tolerate even unbounded reward corruption and can find an O(ε1/4)-optimal policy. We emphasize that FPG is the first that can achieve a meaningful learning guarantee when a constant fraction of episodes are corrupted. Complimentary to the theoretical results, we show that a neural implementation of FPG achieves strong robust learning performance on the MuJoCo continuous control benchmarks.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 48 of its 73 references that are in the map. First posted on arXiv 11 Feb 2021.

Builds on

48 of 73 references.

References · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  7. 2018Is Q-learning Provably Efficient?cited by 485
  8. 2017Robust Adversarial Reinforcement Learningcited by 439
  9. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  10. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  11. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  12. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
36 more
  1. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  2. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  3. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  4. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  5. 2016Agnostic Estimation of Mean and Covariancecited by 131
  6. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  7. 2018Sever: A Robust Meta-Algorithm for Stochastic Optimizationcited by 123
  8. 2018Stochastic bandits robust to adversarial corruptionscited by 118
  9. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  10. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  11. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 105
  12. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  13. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 100
  14. 2019Recent Advances in Algorithmic High-Dimensional Robust Statisticscited by 100
  15. 2019Better Algorithms for Stochastic Bandits with Adversarial Corruptionscited by 91
  16. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  17. 2016Learning from Untrusted Datacited by 90
  18. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  19. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  20. 2019Learning Adversarial MDPs with Bandit Feedback and Unknown Transitioncited by 84
  21. 2017Being Robust (in High Dimensions) Can Be Practicalcited by 83
  22. 2019Online Convex Optimization in Adversarial Markov Decision Processescited by 79
  23. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  24. 2019Policy Poisoning in Batch Reinforcement Learning and Controlcited by 70
  25. 2020Outlier Robust Mean Estimation with Subgaussian Rates via Stabilitycited by 50
  26. 2020Bias no more: high-probability data-dependent regret bounds for adversarial bandits and MDPscited by 49
  27. 2020Simultaneously Learning Stochastic and Adversarial Episodic MDPs with Known Transitioncited by 48
  28. 2016On Lower Bounds for Regret in Reinforcement Learningcited by 39
  29. 2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 39
  30. 2019Variational Regret Bounds for Reinforcement Learningcited by 32
  31. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
  32. 2020A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescited by 32
  33. 2020Online learning in MDPs with linear function approximation and bandit feedbackcited by 29
  34. 2013Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributionscited by 18
  35. 2021Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexitycited by 15
  36. 2019Learning and Planning for Time-Varying MDPs Using Maximum Likelihood Estimationcited by 10

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 10; 2023: 6; 2024: 5; 2025: 6.

30 papers.

Sort:

By area:

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  3. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  4. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  5. 2021A Model Selection Approach for Corruption Robust Reinforcement Learningcited by 45
  6. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  7. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  8. 2023Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attackscited by 14
  9. 2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13
  10. 2022Admissible Policy Teaching through Reward Designcited by 12
  11. 2022Robust Imitation Learning from Corrupted Demonstrationscited by 10
  12. 2022Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learningcited by 9
18 more
  1. 2023Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learningcited by 9
  2. 2024Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedbackcited by 9
  3. 2021Game Redesign in No-regret Game Playingcited by 8
  4. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  5. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  6. 2022Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysiscited by 5
  7. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  8. 2024MetaCURL: Non-stationary Concave Utility Reinforcement Learningcited by 4
  9. 2025Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learningcited by 3
  10. 2023A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agentscited by 2
  11. 2024Robust Thompson Sampling Algorithms Against Reward Poisoning Attackscited by 2
  12. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2
  13. 2023Policy Resilience to Environment Poisoning Attacks on Reinforcement Learningcited by 1
  14. 2023Optimal Cost Constrained Adversarial Attacks For Multiple Agent Systemscited by 1
  15. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  16. 2022User-Oriented Robust Reinforcement Learningcited by 0
  17. 2025Robust In-Context Reinforcement Learning Under Reward Poisoning Attackscited by 0
  18. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0