# Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive

arXiv: https://arxiv.org/abs/2605.00066 · HTML: https://mlsanity.com/paper/2605.00066/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 30 of 30)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023)
- [Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving](https://mlsanity.com/paper/2305.06242/) (2023)
- [End-to-End Driving with Online Trajectory Evaluation via BEV World Model](https://mlsanity.com/paper/2504.01941/) (2025)
- [SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment](https://mlsanity.com/paper/2503.09594/) (2025)
- [DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving](https://mlsanity.com/paper/2308.00398/) (2023)
- [GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.05689/) (2025)
- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning](https://mlsanity.com/paper/2506.06659/) (2025)
- [Bridging Past and Future: End-to-End Autonomous Driving with Historical Prediction and Planning](https://mlsanity.com/paper/2503.14182/) (2025)
- [RAP: 3D Rasterization Augmented End-to-End Planning](https://mlsanity.com/paper/2510.04333/) (2025)
- [Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training](https://mlsanity.com/paper/2503.12030/) (2025)
- [DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving](https://mlsanity.com/paper/2509.17940/) (2025)
- [HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder](https://mlsanity.com/paper/2503.08612/) (2025)
- [SparseDriveV2: Scoring is All You Need for End-to-End Autonomous Driving](https://mlsanity.com/paper/2603.29163/) (2026)
- [LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving](https://mlsanity.com/paper/2512.20563/) (2025)
- [Scaling Laws of Motion Forecasting and Planning -- Technical Report](https://mlsanity.com/paper/2506.08228/) (2025)
- [DiffRefiner: Coarse to Fine Trajectory Planning via Diffusion Refinement with Semantic Interaction for End to End Autonomous Driving](https://mlsanity.com/paper/2511.17150/) (2025)
- [SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse World](https://mlsanity.com/paper/2602.18887/) (2026)

## Built on by (most cited first, 6 of 6)

- [Validate the Dream Before You Trust Its Verdict: Admissibility for World-Model Simulators](https://mlsanity.com/paper/2607.07196/) (2026)
- [When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit](https://mlsanity.com/paper/2608.04896/) (2026)
- [Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms](https://mlsanity.com/paper/2608.20111/) (2026)
- [NavSafe-∞: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments](https://mlsanity.com/paper/2609.26618/) (2026)
- [DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving](https://mlsanity.com/paper/2609.26792/) (2026)
- [When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving](https://mlsanity.com/paper/2609.34440/) (2026)

## Abstract (arXiv)

Open-loop evaluation offers fast, reproducible assessment of autonomous driving planners, but its ability to predict real closed-loop driving performance remains questionable. Prior work has shown that traditional open-loop metrics such as Average Displacement Error (ADE) and Final Displacement Error (FDE) exhibit no reliable correlation with closed-loop Driving Score. In this paper, we ask whether the more recent, safety-aware open-loop metrics introduced by NAVSIM~v2 can bridge this gap. By systematically cross-referencing published results from 15 state-of-the-art methods across NAVSIM (open-loop) and Bench2Drive (closed-loop), we compile a paired dataset of open-loop sub-metrics and closed-loop performance, yielding 8 methods with complete paired data. Our analysis reveals three key findings: (1) the aggregate NAVSIM PDM Score shows a strong positive but non-monotonic correlation with Bench2Drive Driving Score, with clear ranking inversions; (2) among individual NAVSIM sub-metrics, Ego Progress (EP) is the strongest single predictor of closed-loop success, substantially exceeding the safety-critical collision metric NC; (3) the safety-progress trade-off manifests differently in open-loop and closed-loop: methods that maximize safety at the expense of progress rank highly in NAVSIM but underperform in closed-loop due to timeout and slow-driving penalties. We further demonstrate that a much simpler 3-metric formula matches the predictive power of the full 5-metric PDMS at the same Spearman ρ=0.90 on our paired sample of n=8 methods, suggesting that within current state-of-the-art methods -- where TTC and Comfort approach saturation -- these two sub-metrics add little marginal information for closed-loop ranking. Additionally, we identify the snowball effect -- where small open-loop deviations compound into closed-loop failures -- as a candidate mechanism for the residual gap.