mlsanityData to 14 Oct 2026

Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation

arXiv 2506.16753 ↗First posted 20 Jun 2025cs.LG cs.AI cs.ROcited by 0

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

Recently, robust reinforcement learning (RL) methods designed to handle adversarial input observations have received significant attention, motivated by RL's inherent vulnerabilities. While existing approaches have demonstrated reasonable success, addressing worst-case scenarios over long time horizons requires both minimizing the agent's cumulative rewards for adversaries and training agents to counteract them through alternating learning. However, this process introduces mutual dependencies between the agent and the adversary, making interactions with the environment inefficient and hindering the development of off-policy methods. In this work, we propose a novel off-policy method that eliminates the need for additional environmental interactions by reformulating adversarial learning as a soft-constrained optimization problem. Our approach is theoretically supported by the symmetric property of policy evaluation between the agent and the adversary. The implementation is available at https://github.com/nakanakakosuke/VALT_SAC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 66 of its 99 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

66 of 99 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Adversarially Robust Decision Transformer25 Jul
  2. 2024Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents26 Jun
  3. 2024Robust Deep Reinforcement Learning Through Adversarial Attacks and Training : A Survey1 Mar

Other references · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2016Adversarial examples in the physical worldcited by 1,769
  11. 2016OpenAI Gymcited by 1,736
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
38 more
  1. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  2. 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  6. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
  7. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  8. 2017Robust Adversarial Reinforcement Learningcited by 451
  9. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  10. 2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 328
  11. 2017Adversarial Attacks on Neural Network Policiescited by 320
  12. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  13. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  14. 2019Towards Stable and Efficient Training of Verifiably Robust Neural Networkscited by 205
  15. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  16. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  17. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  18. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  19. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  20. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  21. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  22. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  23. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  24. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  25. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 111
  26. 2017Delving into adversarial attacks on deep policiescited by 104
  27. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  28. 2014Algorithms for CVaR Optimization in MDPscited by 99
  29. 2022Robust Reinforcement Learning using Offline Datacited by 96
  30. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  31. 2020Robust Deep Reinforcement Learning through Adversarial Losscited by 75
  32. 2019Policy Poisoning in Batch Reinforcement Learning and Controlcited by 70
  33. 2020Robust Reinforcement Learning using Adversarial Populationscited by 66
  34. 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 63
  35. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
  36. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  37. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  38. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43

50 most cited of 63 All 63 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Robust off-policy Reinforcement Learning via Soft Constrained Adversary
  2. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
  3. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data
  4. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  5. 2023Model-based Offline Policy Optimization with Adversarial Network
  6. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
  7. 2021Offline RL Without Off-Policy Evaluation
  8. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation