# Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.10403 · HTML: https://mlsanity.com/paper/2608.10403/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method](https://mlsanity.com/paper/2003.01197/) (2020)
- [Robust Autonomy Emerges from Self-Play](https://mlsanity.com/paper/2502.03349/) (2025)
- [SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles](https://mlsanity.com/paper/2206.09682/) (2022)
- [CAT: Closed-loop Adversarial Training for Safe End-to-End Driving](https://mlsanity.com/paper/2310.12432/) (2023)
- [Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)](https://mlsanity.com/paper/2505.16394/) (2025)
- [GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS](https://mlsanity.com/paper/2408.01584/) (2024)
- [Driving on Registers](https://mlsanity.com/paper/2601.05083/) (2026)
- [RAP: 3D Rasterization Augmented End-to-End Planning](https://mlsanity.com/paper/2510.04333/) (2025)
- [CaRL: Learning Scalable Planning Policies with Simple Rewards](https://mlsanity.com/paper/2504.17838/) (2025)
- [FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversariality](https://mlsanity.com/paper/2406.02983/) (2024)
- [Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehicles](https://mlsanity.com/paper/2508.14527/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) has shown promising performance in autonomous driving, yet ensuring the safety of online RL policies remains challenging due to insufficient exposure to safety-critical driving scenes. The long-tailed nature of real-world traffic situations makes dangerous and rare interactions difficult to encounter through conventional sampling, limiting the ability of RL policies to learn robust safety behaviors. Existing methods improve training diversity by synthesizing challenging scenes or adversarial situations. However, these approaches typically optimize scene generation objectives separately from the evolving policy, without explicitly modeling how generated perturbations relate to the current policy's weaknesses and learning needs. In this paper, we propose Threat-guided Policy-aware Scene Perturbation (TPSP) for safe autonomous driving with online RL. TPSP introduces a policy-aware scene encoder to capture the interaction between policy behaviors and surrounding environments, enabling scene perturbation aligned with the current policy. Based on this representation, TPSP selectively perturbs critical objects rather than applying uniform modifications across the scene. Furthermore, we develop a threat-guided optimization strategy that evaluates perturbed scenes through threat-level differences between policy rollouts on original and perturbed scenes, guiding the generation of safety-critical scenes with higher training value. Comprehensive experiments demonstrate that TPSP improves safety learning efficiency, achieving strong safety performance on NAVSIM v2 with approximately 4 million kilometers of simulated driving data. Ablation studies verify that policy-aware targeted perturbations provide more informative safety-critical experiences than random or policy-unaware strategies, enabling safer driving under limited interaction budgets.