# When Planners Meet Reality: How Learned, Reactive Traffic Agents Shift nuPlan Benchmarks

arXiv: https://arxiv.org/abs/2510.14677 · HTML: https://mlsanity.com/paper/2510.14677/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Builds on (most cited first, 30 of 31)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [Diffusion-Based Planning for Autonomous Driving with Flexible Guidance](https://mlsanity.com/paper/2501.15564/) (2025)
- [Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients](https://mlsanity.com/paper/2109.13333/) (2021)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [The Waymo Open Sim Agents Challenge](https://mlsanity.com/paper/2305.12032/) (2023)
- [PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving](https://mlsanity.com/paper/2404.14327/) (2024)
- [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018)
- [Trajeglish: Traffic Modeling as Next-Token Prediction](https://mlsanity.com/paper/2312.04535/) (2023)
- [Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world](https://mlsanity.com/paper/2206.09889/) (2022)
- [BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction](https://mlsanity.com/paper/2405.17372/) (2024)
- [DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving](https://mlsanity.com/paper/2310.05885/) (2023)
- [From Prediction to Planning With Goal Conditioned Lane Graph Traversals](https://mlsanity.com/paper/2302.07753/) (2023)
- [Solving Motion Planning Tasks with a Scalable Generative Model](https://mlsanity.com/paper/2407.02797/) (2024)
- [Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?](https://mlsanity.com/paper/2404.07569/) (2024)
- [CaRL: Learning Scalable Planning Policies with Simple Rewards](https://mlsanity.com/paper/2504.17838/) (2025)
- [Versatile Behavior Diffusion for Generalized Traffic Agent Simulation](https://mlsanity.com/paper/2404.02524/) (2024)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023](https://mlsanity.com/paper/2306.11868/) (2023)
- [Large Trajectory Models are Scalable Motion Predictors and Planners](https://mlsanity.com/paper/2310.19620/) (2023)
- [TrafficBots V1.5: Traffic Simulation via Conditional VAEs and Transformers with Relative Pose Encoding](https://mlsanity.com/paper/2406.10898/) (2024)
- [MBAPPE: MCTS-Built-Around Prediction for Planning Explicitly](https://mlsanity.com/paper/2309.08452/) (2023)
- [PseudoMapTrainer: Learning Online Mapping without HD Maps](https://mlsanity.com/paper/2508.18788/) (2025)
- [Pioneering SE(2)-Equivariant Trajectory Planning for Automated Driving](https://mlsanity.com/paper/2403.11304/) (2024)
- [Learning Through Retrospection: Improving Trajectory Prediction for Automated Driving with Error Feedback](https://mlsanity.com/paper/2504.13785/) (2025)

## Built on by (most cited first, 7 of 7)

- [Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Response](https://mlsanity.com/paper/2512.03936/) (2025)
- [DiffuSearch: How Hybrid Trajectory Planning Benefits from Aligned Objectives in Diffusion and Action Space](https://mlsanity.com/paper/2609.02252/) (2026)
- [Spark: Modular Spiking Neural Networks](https://mlsanity.com/paper/2602.02306/) (2026)
- [Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos](https://mlsanity.com/paper/2606.07366/) (2026)
- [Shift & Drift: A Zero-Shot Benchmark for Generalizable and Robust Autonomous Driving Motion Planning](https://mlsanity.com/paper/2607.07844/) (2026)
- [Real-Time Rulebook-Aware Nonlinear MPC for Autonomous Driving with Priority-Biased Tiered Slacks](https://mlsanity.com/paper/2607.10975/) (2026)
- [TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving](https://mlsanity.com/paper/2609.38463/) (2026)

## Abstract (arXiv)

Planner evaluation in closed-loop simulation often uses rule-based traffic agents, whose simplistic and passive behavior can hide planner deficiencies and bias rankings. Widely used IDM agents simply follow a lead vehicle and cannot react to vehicles in adjacent lanes, hindering tests of complex interaction capabilities. We address this issue by integrating the state-of-the-art learned traffic agent model SMART into nuPlan. Thus, we are the first to evaluate planners under more realistic conditions and quantify how conclusions shift when narrowing the sim-to-real gap. Our analysis covers 14 recent planners and established baselines and shows that IDM-based simulation overestimates planning performance: nearly all scores deteriorate. In contrast, many planners interact better than previously assumed and even improve in multi-lane, interaction-heavy scenarios like lane changes or turns. Methods trained in closed-loop demonstrate the best and most stable driving performance. However, when reaching their limits in augmented edge-case scenarios, all learned planners degrade abruptly, whereas rule-based planners maintain reasonable basic behavior. Based on our results, we suggest SMART-reactive simulation as a new standard closed-loop benchmark in nuPlan and release the SMART agents as a drop-in alternative to IDM at https://github.com/shgd95/InteractiveClosedLoop.