# ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling

arXiv: https://arxiv.org/abs/2306.12241 · HTML: https://mlsanity.com/paper/2306.12241/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2023.

## Built on by (most cited first, 30 of 60)

- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.15208/) (2024)
- [UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction](https://mlsanity.com/paper/2403.15098/) (2024)
- [RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios](https://mlsanity.com/paper/2312.13303/) (2023)
- [GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction](https://mlsanity.com/paper/2402.16174/) (2024)
- [SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction](https://mlsanity.com/paper/2403.11492/) (2024)
- [SimGen: Simulator-conditioned Driving Scene Generation](https://mlsanity.com/paper/2406.09386/) (2024)
- [CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2502.19908/) (2025)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving](https://mlsanity.com/paper/2409.18343/) (2024)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [Rethinking Imitation-based Planner for Autonomous Driving](https://mlsanity.com/paper/2309.10443/) (2023)
- [Probing Multimodal LLMs as World Models for Driving](https://mlsanity.com/paper/2405.05956/) (2024)
- [Embodied Scene Understanding for Vision Language Models via MetaVQA](https://mlsanity.com/paper/2501.09167/) (2025)
- [Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test](https://mlsanity.com/paper/2503.02911/) (2025)
- [Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model](https://mlsanity.com/paper/2409.09575/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Extrapolated Urban View Synthesis Benchmark](https://mlsanity.com/paper/2412.05256/) (2024)
- [Unraveling the Effects of Synthetic Data on End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.18108/) (2025)
- [Decoupled Diffusion Sparks Adaptive Scene Generation](https://mlsanity.com/paper/2504.10485/) (2025)
- [CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectories](https://mlsanity.com/paper/2403.13208/) (2024)
- [TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Autonomous Driving](https://mlsanity.com/paper/2404.02410/) (2024)
- [CooPre: Cooperative Pretraining for V2X Cooperative Perception](https://mlsanity.com/paper/2408.11241/) (2024)
- [BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.10856/) (2026)
- [LASIL: Learner-Aware Supervised Imitation Learning For Long-term Microscopic Traffic Simulation](https://mlsanity.com/paper/2403.17601/) (2024)
- [Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models](https://mlsanity.com/paper/2406.04300/) (2024)
- [Projected Coupled Diffusion for Test-Time Constrained Joint Generation](https://mlsanity.com/paper/2508.10531/) (2025)
- [ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2508.07089/) (2025)
- [UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos](https://mlsanity.com/paper/2510.15018/) (2025)
- [Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving](https://mlsanity.com/paper/2605.10034/) (2026)

## Abstract (arXiv)

Large-scale driving datasets such as Waymo Open Dataset and nuScenes substantially accelerate autonomous driving research, especially for perception tasks such as 3D detection and trajectory forecasting. Since the driving logs in these datasets contain HD maps and detailed object annotations which accurately reflect the real-world complexity of traffic behaviors, we can harvest a massive number of complex traffic scenarios and recreate their digital twins in simulation. Compared to the hand-crafted scenarios often used in existing simulators, data-driven scenarios collected from the real world can facilitate many research opportunities in machine learning and autonomous driving. In this work, we present ScenarioNet, an open-source platform for large-scale traffic scenario modeling and simulation. ScenarioNet defines a unified scenario description format and collects a large-scale repository of real-world traffic scenarios from the heterogeneous data in various driving datasets including Waymo, nuScenes, Lyft L5, and nuPlan datasets. These scenarios can be further replayed and interacted with in multiple views from Bird-Eye-View layout to realistic 3D rendering in MetaDrive simulator. This provides a benchmark for evaluating the safety of autonomous driving stacks in simulation before their real-world deployment. We further demonstrate the strengths of ScenarioNet on large-scale scenario generation, imitation learning, and reinforcement learning in both single-agent and multi-agent settings. Code, demo videos, and website are available at https://metadriverse.github.io/scenarionet.