On Minimizing Adversarial Counterfactual Error in Adversarial RL
Not on the map. Why?
Abstract (arXiv)
Deep Reinforcement Learning (DRL) policies are highly susceptible to adversarial noise in observations, which poses significant risks in safety-critical scenarios. The challenge inherent to adversarial perturbations is that by altering the information observed by the agent, the state becomes only partially observable. Existing approaches address this by either enforcing consistent actions across nearby states or maximizing the worst-case value within adversarially perturbed observations. However, the former suffers from performance degradation when attacks succeed, while the latter tends to be overly conservative, leading to suboptimal performance in benign settings. We hypothesize that these limitations stem from their failing to account for partial observability directly. To this end, we introduce a novel objective called Adversarial Counterfactual Error (ACoE), defined on the beliefs about the true state and balancing value optimization with robustness. To make ACoE scalable in model-free settings, we propose the theoretically-grounded surrogate objective Cumulative-ACoE (C-ACoE). Our empirical evaluations on standard benchmarks (MuJoCo, Atari, and Highway) demonstrate that our method significantly outperforms current state-of-the-art approaches for addressing adversarial RL challenges, offering a promising direction for improving robustness in DRL under adversarial conditions. Our code is available at https://github.com/romanbelaire/acoe-robust-rl.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 7 Jun 2024.
Builds on
41 of 56 references.
References · 41
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2015Domain-Adversarial Training of Neural Networkscited by 3,023
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2020Fast is better than free: Revisiting adversarial trainingcited by 723
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2019Adversarial Training for Free!cited by 638
- 2017Robust Adversarial Reinforcement Learningcited by 439
- 2017Adversarial Attacks on Neural Network Policiescited by 314
- 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
- 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
29 more
- 2020Understanding and Improving Fast Adversarial Trainingcited by 213
- 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
- 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
- 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
- 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
- 2017Delving into adversarial attacks on deep policiescited by 104
- 2018Universal Adversarial Trainingcited by 85
- 2020Robust Deep Reinforcement Learning through Adversarial Losscited by 73
- 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 62
- 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 54
- 2019Certified Adversarial Robustness for Deep Reinforcement Learningcited by 47
- 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
- 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 42
- 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
- 2019Model-Based Reinforcement Learning with Adversarial Training for Online Recommendationcited by 33
- 2020Robustifying Reinforcement Learning Agents via Action Space Adversarial Trainingcited by 30
- 2019Transfer of Adversarial Robustness Between Perturbation Typescited by 25
- 2023Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustnesscited by 18
- 2022Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculumcited by 15
- 2017Regret Minimization for Partially Observable Deep Reinforcement Learningcited by 12
- 2020Solving Zero-Sum One-Sided Partially Observable Stochastic Gamescited by 12
- 2023Optimal Attack and Defense for Reinforcement Learningcited by 12
- 2020Minimax Regret Optimisation for Robust Planning in Uncertain Markov Decision Processescited by 10
- 2021The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPscited by 9
- 2024Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policiescited by 8
- 2023Improve Robustness of Reinforcement Learning against Observation Perturbations via l_∞ Lipschitz Policy Networkscited by 7
- 2024Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-errorcited by 4
- 2023Regret-Based Defense in Adversarial Reinforcement Learningcited by 3
- 2023Certified Robust Control under Adversarial Perturbationscited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.