# Sparse Threats, Focused Defense: Criticality-Aware Robust Reinforcement Learning for Safe Autonomous Driving

arXiv: https://arxiv.org/abs/2601.01800 · HTML: https://mlsanity.com/paper/2601.01800/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving.
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Placed in

- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)
- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving](https://mlsanity.com/paper/1905.02680/) (2019)
- [VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving](https://mlsanity.com/paper/2412.15544/) (2024)
- [HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving](https://mlsanity.com/paper/2401.03160/) (2024)
- [Characterizing Attacks on Deep Reinforcement Learning](https://mlsanity.com/paper/1907.09470/) (2019)
- [Limitations and Improvements of the Intelligent Driver Model (IDM)](https://mlsanity.com/paper/2104.02583/) (2021)
- [Research on Autonomous Driving Decision-making Strategies based Deep Reinforcement Learning](https://mlsanity.com/paper/2408.03084/) (2024)
- [Targeted Attack on Deep RL-based Autonomous Driving with Learned Visual Patterns](https://mlsanity.com/paper/2109.07723/) (2021)
- [Robust Deep Reinforcement Learning Through Adversarial Attacks and Training : A Survey](https://mlsanity.com/paper/2403.00420/) (2024)
- [On Data Fabrication in Collaborative Vehicular Perception: Attacks and Countermeasures](https://mlsanity.com/paper/2309.12955/) (2023)
- [SoK: Rethinking Sensor Spoofing Attacks against Robotic Vehicles from a Systematic View](https://mlsanity.com/paper/2205.04662/) (2022)
- [Steerable Adversarial Scenario Generation through Test-Time Preference Alignment](https://mlsanity.com/paper/2509.20102/) (2025)
- [Data Poisoning Attacks in Intelligent Transportation Systems: A Survey](https://mlsanity.com/paper/2407.15855/) (2024)
- [Sharpening the Spear: Adaptive Expert-Guided Adversarial Attack Against DRL-based Autonomous Driving Policies](https://mlsanity.com/paper/2506.18304/) (2025)
- [Enhancing Security in Deep Reinforcement Learning: A Comprehensive Survey on Adversarial Attacks and Defenses](https://mlsanity.com/paper/2510.20314/) (2025)
- [Robust Driving Control for Autonomous Vehicles: An Intelligent General-sum Constrained Adversarial Reinforcement Learning Approach](https://mlsanity.com/paper/2510.09041/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) has shown considerable potential in autonomous driving (AD), yet its vulnerability to perturbations remains a critical barrier to real-world deployment. As a primary countermeasure, adversarial training improves policy robustness by training the AD agent in the presence of an adversary that deliberately introduces perturbations. Existing approaches typically model the interaction as a zero-sum game with continuous attacks. However, such designs overlook the inherent asymmetry between the agent and the adversary and then fail to reflect the sparsity of safety-critical risks, rendering the achieved robustness inadequate for practical AD scenarios. To address these limitations, we introduce criticality-aware robust RL (CARRL), a novel adversarial training approach for handling sparse, safety-critical risks in autonomous driving. CARRL consists of two interacting components: a risk exposure adversary (REA) and a risk-targeted robust agent (RTRA). We model the interaction between the REA and RTRA as a general-sum game, allowing the REA to focus on exposing safety-critical failures (e.g., collisions) while the RTRA learns to balance safety with driving efficiency. The REA employs a decoupled optimization mechanism to better identify and exploit sparse safety-critical moments under a constrained budget. However, such focused attacks inevitably result in a scarcity of adversarial data. The RTRA copes with this scarcity by jointly leveraging benign and adversarial experiences via a dual replay buffer and enforces policy consistency under perturbations to stabilize behavior. Experimental results demonstrate that our approach reduces the collision rate by at least 22.66% across all cases compared to state-of-the-art baseline methods.