# Online Robust Policy Learning in the Presence of Unknown Adversaries

arXiv: https://arxiv.org/abs/1807.06064 · HTML: https://mlsanity.com/paper/1807.06064/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2018.

## Built on by (most cited first, 15 of 15)

- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Robust Adversarial Attacks Detection based on Explainable Deep Reinforcement Learning For UAV Guidance and Planning](https://mlsanity.com/paper/2206.02670/) (2022)
- [Federated Learning and Meta Learning: Approaches, Applications, and Directions](https://mlsanity.com/paper/2210.13111/) (2022)
- [Resilient Machine Learning for Networked Cyber Physical Systems: A Survey for Machine Learning Security to Securing Machine Learning for CPS](https://mlsanity.com/paper/2102.07244/) (2021)
- [Defending Observation Attacks in Deep Reinforcement Learning via Detection and Denoising](https://mlsanity.com/paper/2206.07188/) (2022)
- [Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies](https://mlsanity.com/paper/2402.12673/) (2024)
- [Targeted Adversarial Attacks on Deep Reinforcement Learning Policies via Model Checking](https://mlsanity.com/paper/2212.05337/) (2022)
- [Robust Communicative Multi-Agent Reinforcement Learning with Active Defense](https://mlsanity.com/paper/2312.11545/) (2023)
- [Fast Certification of Vision-Language Models Using Incremental Randomized Smoothing](https://mlsanity.com/paper/2311.09024/) (2023)
- [Benchmarking Deep Reinforcement Learning for Navigation in Denied Sensor Environments](https://mlsanity.com/paper/2410.14616/) (2024)
- [Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theorems](https://mlsanity.com/paper/2303.12981/) (2023)
- [Robust Reinforcement Learning under model misspecification](https://mlsanity.com/paper/2103.15370/) (2021)
- [Security of Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2212.06123/) (2022)
- [Training RL Agents for Multi-Objective Network Defense Tasks](https://mlsanity.com/paper/2505.22531/) (2025)
- [Neutral Agent-based Adversarial Policy Learning against Deep Reinforcement Learning in Multi-party Open Systems](https://mlsanity.com/paper/2510.10937/) (2025)

## Abstract (arXiv)

The growing prospect of deep reinforcement learning (DRL) being used in cyber-physical systems has raised concerns around safety and robustness of autonomous agents. Recent work on generating adversarial attacks have shown that it is computationally feasible for a bad actor to fool a DRL policy into behaving sub optimally. Although certain adversarial attacks with specific attack models have been addressed, most studies are only interested in off-line optimization in the data space (e.g., example fitting, distillation). This paper introduces a Meta-Learned Advantage Hierarchy (MLAH) framework that is attack model-agnostic and more suited to reinforcement learning, via handling the attacks in the decision space (as opposed to data space) and directly mitigating learned bias introduced by the adversary. In MLAH, we learn separate sub-policies (nominal and adversarial) in an online manner, as guided by a supervisory master agent that detects the presence of the adversary by leveraging the advantage function for the sub-policies. We demonstrate that the proposed algorithm enables policy learning with significantly lower bias as compared to the state-of-the-art policy learning approaches even in the presence of heavy state information attacks. We present algorithm analysis and simulation results using popular OpenAI Gym environments.