Data to 7 Oct 2026

Adversarial Policies: Attacking Deep Reinforcement Learning

arXiv 1905.10615 ↗First posted 25 May 2019cs.LG cs.AI cs.CRcited by 209

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.

Key facts. Cited by 209 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 43; 2022: 36; 2023: 39; 2024: 26; 2025: 33; 2026: 31.

209 papers.

Sort:

By area:

2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2021Unsolved Problems in ML Safetycited by 291
2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 66
2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
2022X-Risk Analysis for AI Researchcited by 60
2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 54

38 more

2022Sound Adversarial Audio-Visual Navigationcited by 46
2024Evaluating Language Model Agency through Negotiationscited by 43
2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 42
2021CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothingcited by 41
2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 39
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
2024Reward-Robust RLHF in LLMscited by 32
2022Adversarial Policies Beat Superhuman Go AIscited by 29
2023BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learningcited by 29
2021Evaluating the Robustness of Collaborative Agentscited by 25
2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
2022Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learningcited by 21
2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 19
2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
2024SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agentscited by 19
2025RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?cited by 19
2021Center Smoothing: Certified Robustness for Networks with Structured Outputscited by 18
2021Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learningcited by 18
2022PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learningcited by 18
2023Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustnesscited by 18
2021ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxationcited by 17
2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
2024Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policiescited by 17
2022Adversarial Cheap Talkcited by 16
2023Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influencecited by 16
2024RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviorscited by 16
2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15
2021Investigating Vulnerabilities of Deep Neural Policiescited by 15
2021Universal Spectral Adversarial Attacks for Deformable Shapescited by 14
2021Neural Network Verification in Controlcited by 14
2022Are AlphaZero-like Agents Robust to Adversarial Perturbations?cited by 14
2023Implicit Poisoning Attacks in Two-Agent Reinforcement Learning: Adversarial Policies for Training-Time Attackscited by 14
2021Multi-Task Federated Reinforcement Learning with Adversariescited by 13
2021Targeted Attack on Deep RL-based Autonomous Driving with Learned Visual Patternscited by 13
2023Overload: Latency Attacks on Object Detection for Edge Devicescited by 13
2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 11

50 most cited of 209 All 209 (JSON)