# Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives arXiv: https://arxiv.org/abs/2605.19771 · HTML: https://mlsanity.com/paper/2605.19771/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. - Builds on 47 of its 52 references that are in the map. - First posted on arXiv 19 May 2026. ## Placed in - End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/) - Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/) ## Builds on (most cited first, 30 of 47) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022) - [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022) - [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023) - [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021) - [Mean Flows for One-step Generative Modeling](https://mlsanity.com/paper/2505.13447/) (2025) - [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021) - [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021) - [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024) - [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024) - [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024) - [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025) - [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024) - [PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving](https://mlsanity.com/paper/2112.12610/) (2021) - [Enhancing End-to-End Autonomous Driving with Latent World Model](https://mlsanity.com/paper/2406.08481/) (2024) - [Flow Matching in Latent Space](https://mlsanity.com/paper/2307.08698/) (2023) - [End-to-End Driving with Online Trajectory Evaluation via BEV World Model](https://mlsanity.com/paper/2504.01941/) (2025) - [Pseudo-Simulation for Autonomous Driving](https://mlsanity.com/paper/2506.04218/) (2025) - [Epona: Autoregressive Diffusion World Model for Autonomous Driving](https://mlsanity.com/paper/2506.24113/) (2025) - [DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers](https://mlsanity.com/paper/2412.18607/) (2024) - [DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning](https://mlsanity.com/paper/2506.06659/) (2025) - [HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving](https://mlsanity.com/paper/2412.01718/) (2024) - [ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Driving](https://mlsanity.com/paper/2504.19580/) (2025) - [DriveLaW:Unifying Planning and Video Generation in a Latent Driving World](https://mlsanity.com/paper/2512.23421/) (2025) ## Built on by (most cited first, 8 of 8) - [DA-WAM: Decision-Aligned Future Latents for Driving World Models](https://mlsanity.com/paper/2608.19085/) (2026) - [DriveZero: End-to-End Driving Beyond Human Demonstrations](https://mlsanity.com/paper/2609.06055/) (2026) - [ReWorld: Representation Learning for World Action Models](https://mlsanity.com/paper/2606.27504/) (2026) - [SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models](https://mlsanity.com/paper/2610.13781/) (2026) - [DriveReferee: Geometric Safety Verdicts Need Not Be Learned for Driving World-Action Models](https://mlsanity.com/paper/2609.22762/) (2026) - [NavSafe-∞: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments](https://mlsanity.com/paper/2609.26618/) (2026) - [PhysWAM: Physically Consistent World Action Model for Autonomous Driving](https://mlsanity.com/paper/2609.37970/) (2026) - [Beyond Waypoint Regression: Query-Based Cost Learning over Reachable Ego Futures for End-to-End Driving](https://mlsanity.com/paper/2610.08123/) (2026) ## Abstract (arXiv) Existing imitation learning methods for end-to-end autonomous driving predominantly learn from successful demonstrations by minimizing geometric deviations from expert trajectories. This paradigm implicitly assumes that spatial proximity implies behavioral safety, leading to a critical objective mismatch: trajectories with nearly identical imitation losses may exhibit drastically different safety outcomes, where one remains recoverable while the other results in collision. To address this limitation, we propose BeyondDrive, a failure-aware imitation learning framework that jointly learns from successful and failed driving behaviors. First, we introduce a flow matching-based negative trajectory generator that synthesizes safety-critical yet expert-proximate trajectories, enabling explicit modeling of safety asymmetry. Second, we develop a diversity-aware sampling strategy that mitigates mode collapse and improves coverage of diverse failure modes during negative trajectory generation. Third, we propose a Repulsive Distance Loss that simultaneously attracts predictions toward expert demonstrations while repelling them from hard negative trajectories, thereby establishing discriminative safety boundaries in trajectory space. Applied to the uni-modal baseline Latent TransFuser, BeyondDrive achieves 89.7 PDMS on the NAVSIMv1 closed-loop benchmark, outperforming prior state-of-the-art methods. Moreover, BeyondDrive generalizes effectively across different autonomous driving architectures, including multi-modal planners, and further demonstrates strong zero-shot transferability on the HUGSIM benchmark.