mlsanityData to 14 Oct 2026

On the Tension Between Optimality and Adversarial Robustness in Policy Optimization

arXiv 2512.01228 ↗First posted 1 Dec 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

Achieving optimality and adversarial robustness in deep reinforcement learning has long been regarded as conflicting goals. Nonetheless, recent theoretical insights presented in CAR suggest a potential alignment, raising the important question of how to realize this in practice. This paper first identifies a key gap between theory and practice by comparing standard policy optimization (SPO) and adversarially robust policy optimization (ARPO). Although they share theoretical consistency, a fundamental tension between robustness and optimality arises in practical policy gradient methods. SPO tends toward convergence to vulnerable first-order stationary policies (FOSPs) with strong natural performance, whereas ARPO typically favors more robust FOSPs at the expense of reduced returns. Furthermore, we attribute this tradeoff to the reshaping effect of the strongest adversary in ARPO, which significantly complicates the global landscape by inducing deceptive sticky FOSPs. This improves robustness but makes navigation more challenging. To alleviate this, we develop the BARPO, a bilevel framework unifying SPO and ARPO by modulating adversary strength, thereby facilitating navigability while preserving global optima. Extensive empirical results demonstrate that BARPO consistently outperforms vanilla ARPO, providing a practical approach to reconcile theoretical and empirical performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 72 of its 101 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

72 of 101 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Towards Optimal Adversarial Robust Reinforcement Learning with Infinity Measurement Error23 Feb
  2. 2024Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents26 Jun
  3. 2024On Minimizing Adversarial Counterfactual Error in Adversarial RL7 Jun
  4. 2024Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies20 Feb
  5. 2024Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error3 Feb

Other references · 67

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  11. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  12. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
38 more
  1. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  3. 2017Robust Adversarial Reinforcement Learningcited by 451
  4. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  5. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  6. 2017Adversarial Attacks on Neural Network Policiescited by 320
  7. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  8. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  9. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  10. 2019Towards Stable and Efficient Training of Verifiably Robust Neural Networkscited by 205
  11. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 197
  12. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  13. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  14. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  15. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  16. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  17. 2017Delving into adversarial attacks on deep policiescited by 104
  18. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
  19. 2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
  20. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  21. 2020Robust Deep Reinforcement Learning through Adversarial Losscited by 75
  22. 2023Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveyscited by 75
  23. 2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 68
  24. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  25. 2020Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learningcited by 61
  26. 2016Error bounds, quadratic growth, and linear convergence of proximal methodscited by 60
  27. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
  28. 2022Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspectivecited by 51
  29. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  30. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  31. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43
  32. 2021CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothingcited by 42
  33. 2023First-order penalty methods for bilevel optimizationcited by 41
  34. 2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 40
  35. 2017First Order Methods beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problemscited by 37
  36. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 36
  37. 2017Whatever Does Not Kill Deep Reinforcement Learning, Makes It Strongercited by 35
  38. 2019The Value Function Polytope in Reinforcement Learningcited by 29

50 most cited of 67 All 67 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Robust Adversarial Policy Optimization Under Dynamics Uncertainty
  2. 2026Robust Policy Optimization via Adversarial Importance Sampling
  3. 2026Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization
  4. 2026RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning
  5. 2024Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets
  6. 2025Adversarial Reinforcement Learning: A Duality-Based Approach To Solving Optimal Control Problems
  7. 2025Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learning
  8. 2026ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving