# Training Adversarial Agents to Exploit Weaknesses in Deep Control Policies

arXiv: https://arxiv.org/abs/2002.12078 · HTML: https://mlsanity.com/paper/2002.12078/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2020.

## Built on by (most cited first, 24 of 24)

- [A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation](https://mlsanity.com/paper/2112.00964/) (2021)
- [Adversarial Training is Not Ready for Robot Learning](https://mlsanity.com/paper/2103.08187/) (2021)
- [Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles](https://mlsanity.com/paper/2109.06126/) (2021)
- [Finding Critical Scenarios for Automated Driving Systems: A Systematic Literature Review](https://mlsanity.com/paper/2110.08664/) (2021)
- [Detecting Multi-Sensor Fusion Errors in Advanced Driver-Assistance Systems](https://mlsanity.com/paper/2109.06404/) (2021)
- [VILTA: A VLM-in-the-Loop Adversary for Enhancing Driving Policy Robustness](https://mlsanity.com/paper/2601.12672/) (2026)
- [Adversarial Mixture Density Networks: Learning to Drive Safely from Collision Data](https://mlsanity.com/paper/2107.04485/) (2021)
- [Self-Improving Safety Performance of Reinforcement Learning Based Driving with Black-Box Verification Algorithms](https://mlsanity.com/paper/2210.16575/) (2022)
- [Safety-Critical Traffic Simulation with Adversarial Transfer of Driving Intentions](https://mlsanity.com/paper/2503.05180/) (2025)
- [ARC: Adversarially Robust Control Policies for Autonomous Vehicles](https://mlsanity.com/paper/2107.04487/) (2021)
- [ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving](https://mlsanity.com/paper/2603.15221/) (2026)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [ECSAS: Exploring Critical Scenarios from Action Sequence in Autonomous Driving](https://mlsanity.com/paper/2209.10078/) (2022)
- [A Deep Reinforcement Learning Approach to Rare Event Estimation](https://mlsanity.com/paper/2211.12470/) (2022)
- [AED: Automatic Discovery of Effective and Diverse Vulnerabilities for Autonomous Driving Policy with Large Language Models](https://mlsanity.com/paper/2503.20804/) (2025)
- [RADE: Learning Risk-Adjustable Driving Environment via Multi-Agent Conditional Diffusion](https://mlsanity.com/paper/2505.03178/) (2025)
- [Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages](https://mlsanity.com/paper/2103.09726/) (2021)
- [Multi-Agent Vulnerability Discovery for Autonomous Driving with Hazard Arbitration Reward](https://mlsanity.com/paper/2112.06185/) (2021)
- [Driving in Corner Case: A Real-World Adversarial Closed-Loop Evaluation Platform for End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.16055/) (2025)
- [Training Adversarial yet Safe Agent to Characterize Safety Performance of Highly Automated Vehicles](https://mlsanity.com/paper/2402.01576/) (2024)
- [Dynamically Expanding Capacity of Autonomous Driving with Near-Miss Focused Training Framework](https://mlsanity.com/paper/2406.02865/) (2024)
- [Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Traffic](https://mlsanity.com/paper/2604.12857/) (2026)
- [ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving](https://mlsanity.com/paper/2605.21168/) (2026)
- [Teach-to-Crash: A Closed-Loop Student-Teacher LLM Framework for Collision-Inducing Test Scenario Generation](https://mlsanity.com/paper/2609.27296/) (2026)

## Abstract (arXiv)

Deep learning has become an increasingly common technique for various control problems, such as robotic arm manipulation, robot navigation, and autonomous vehicles. However, the downside of using deep neural networks to learn control policies is their opaque nature and the difficulties of validating their safety. As the networks used to obtain state-of-the-art results become increasingly deep and complex, the rules they have learned and how they operate become more challenging to understand. This presents an issue, since in safety-critical applications the safety of the control policy must be ensured to a high confidence level. In this paper, we propose an automated black box testing framework based on adversarial reinforcement learning. The technique uses an adversarial agent, whose goal is to degrade the performance of the target model under test. We test the approach on an autonomous vehicle problem, by training an adversarial reinforcement learning agent, which aims to cause a deep neural network-driven autonomous vehicle to collide. Two neural networks trained for autonomous driving are compared, and the results from the testing are used to compare the robustness of their learned control policies. We show that the proposed framework is able to find weaknesses in both control policies that were not evident during online testing and therefore, demonstrate a significant benefit over manual testing methods.