# ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving

arXiv: https://arxiv.org/abs/2605.21168 · HTML: https://mlsanity.com/paper/2605.21168/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving.
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [Adversarial Training Can Hurt Generalization](https://mlsanity.com/paper/1906.06032/) (2019)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients](https://mlsanity.com/paper/2204.13683/) (2022)
- [ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles](https://mlsanity.com/paper/2405.14062/) (2024)
- [On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2201.05057/) (2022)
- [Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method](https://mlsanity.com/paper/2003.01197/) (2020)
- [SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles](https://mlsanity.com/paper/2206.09682/) (2022)
- ["Curse of rarity" for autonomous vehicles](https://mlsanity.com/paper/2207.02749/) (2022)
- [Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation](https://mlsanity.com/paper/2009.08311/) (2020)
- [Safe Reinforcement Learning Using Advantage-Based Intervention](https://mlsanity.com/paper/2106.09110/) (2021)
- [Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarios](https://mlsanity.com/paper/2004.06531/) (2020)
- [AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms](https://mlsanity.com/paper/1901.07849/) (2019)
- [Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving](https://mlsanity.com/paper/1903.10654/) (2019)
- [Towards Safe Reinforcement Learning with a Safety Editor Policy](https://mlsanity.com/paper/2201.12427/) (2022)
- [Training Adversarial Agents to Exploit Weaknesses in Deep Control Policies](https://mlsanity.com/paper/2002.12078/) (2020)
- [Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles](https://mlsanity.com/paper/2103.07403/) (2021)
- [Safety-Critical Scenario Generation Via Reinforcement Learning Based Editing](https://mlsanity.com/paper/2306.14131/) (2023)
- [Fundamental Considerations around Scenario-Based Testing for Automated Driving](https://mlsanity.com/paper/2005.04045/) (2020)
- [Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors](https://mlsanity.com/paper/2408.03200/) (2024)
- [FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversariality](https://mlsanity.com/paper/2406.02983/) (2024)
- [NEST: A Neuromodulated Small-world Hypergraph Trajectory Prediction Model for Autonomous Driving](https://mlsanity.com/paper/2412.11682/) (2024)
- [Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Model](https://mlsanity.com/paper/2412.00243/) (2024)

## Abstract (arXiv)

Safety-critical scenarios are central to evaluating autonomous driving systems, yet their rarity in naturalistic logs makes simulation-based stress testing indispensable. Most scenario generation methods treat surrounding agents as adversaries, but they either (i) induce failures without explicitly modeling vehicle-road physical limits, yielding visually extreme yet physically unsolvable crashes, or (ii) enforce physical feasibility or policy feasibility in isolation, which can over-focus on aggressive maneuvers or remain tied to a controller-dependent capability boundary. We propose ScenePilot, a feasibility-guided, boundary-driven framework that targets the boundary band: scenarios that are physically solvable in principle yet still cause the deployed autonomy stack to fail. We formulate generation as constrained multi-objective reinforcement learning, combining an RSS-derived physical-feasibility score σ with an online-learned AV-risk predictor Phi, and introduce step-level feasibility-aware shielding to keep exploration near the feasibility boundary while avoiding infeasible artifacts. Experiments on SafeBench with multiple planners show that ScenePilot yields substantially higher collision rates (+6.2 percentage points) while preserving physical validity, and that adversarial fine-tuning on these boundary-band scenarios consistently reduces downstream crash rates. The code is available at https://github.com/QiyuRuan/ScenePilot.