Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error

arXiv 2402.02165 ↗First posted 3 Feb 2024cs.LGcited by 4

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

Establishing robust policies is essential to counter attacks or disturbances affecting deep reinforcement learning (DRL) agents. Recent studies explore state-adversarial robustness and suggest the potential lack of an optimal robust policy (ORP), posing challenges in setting strict robustness constraints. This work further investigates ORP: At first, we introduce a consistency assumption of policy (CAP) stating that optimal actions in the Markov decision process remain consistent with minor perturbations, supported by empirical and theoretical evidence. Building upon CAP, we crucially prove the existence of a deterministic and stationary ORP that aligns with the Bellman optimal policy. Furthermore, we illustrate the necessity of L∞-norm when minimizing Bellman error to attain ORP. This finding clarifies the vulnerability of prior DRL algorithms that target the Bellman optimal policy with L1-norm and motivates us to train a Consistent Adversarial Robust Deep Q-Network (CAR-DQN) by minimizing a surrogate of Bellman Infinity-error. The top-tier performance of CAR-DQN across various benchmarks validates its practical effectiveness and reinforces the soundness of our theoretical analysis.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 37 of its 44 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

37 of 44 references.

References · 37

2017Proximal Policy Optimization Algorithmscited by 15,215
2014Explaining and Harnessing Adversarial Examplescited by 7,364
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
2015Continuous control with deep reinforcement learningcited by 3,407
2015Trust Region Policy Optimizationcited by 2,580
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2016OpenAI Gymcited by 1,736
2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 328
2017Adversarial Attacks on Neural Network Policiescited by 320

25 more

2019Reinforcement Learning in Healthcare: A Surveycited by 310
2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
2019Towards Stable and Efficient Training of Verifiably Robust Neural Networkscited by 205
2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
2017Delving into adversarial attacks on deep policiescited by 104
2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
2020Robust Deep Reinforcement Learning through Adversarial Losscited by 75
2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 68
2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
2022Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspectivecited by 51
2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43
2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 36
2017Whatever Does Not Kill Deep Reinforcement Learning, Makes It Strongercited by 35
2019Online Robustness Training for Deep Reinforcement Learningcited by 26
2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
2022PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learningcited by 19
2023Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustnesscited by 19
2022Provable Defense against Backdoor Policies in Reinforcement Learningcited by 18
2019Snooping Attacks on Deep Reinforcement Learningcited by 17
2022Adversarial Cheap Talkcited by 16
2023Improve Robustness of Reinforcement Learning against Observation Perturbations via l_∞ Lipschitz Policy Networkscited by 7

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

2024Robust Deep Reinforcement Learning against Adversarial Behavior Manipulationcited by 2
2024On Minimizing Adversarial Counterfactual Error in Adversarial RLcited by 2
2025Diffusion Guided Adversarial State Perturbations in Reinforcement Learningcited by 0
2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way