# From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving

arXiv: https://arxiv.org/abs/2305.13960 · HTML: https://mlsanity.com/paper/2305.13960/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 69 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 37)

- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering](https://mlsanity.com/paper/2112.05504/) (2021)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [DriveGAN: Towards a Controllable High-Quality Neural Simulation](https://mlsanity.com/paper/2104.15060/) (2021)
- [CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario](https://mlsanity.com/paper/1905.05217/) (2019)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [SceneGen: Learning to Generate Realistic Traffic Scenes](https://mlsanity.com/paper/2101.06541/) (2021)
- [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021)
- [Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs](https://mlsanity.com/paper/1805.05499/) (2018)
- [SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving](https://mlsanity.com/paper/2005.03844/) (2020)
- [Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation](https://mlsanity.com/paper/2008.09092/) (2020)
- [Scenic: A Language for Scenario Specification and Data Generation](https://mlsanity.com/paper/2010.06580/) (2020)
- [Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving](https://mlsanity.com/paper/1903.10654/) (2019)
- [Adversarial Reinforcement Learning for Procedural Content Generation](https://mlsanity.com/paper/2103.04847/) (2021)
- [Learning Camera-Aware Noise Models](https://mlsanity.com/paper/2008.09370/) (2020)
- [Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art](https://mlsanity.com/paper/2108.02403/) (2021)

## Built on by (most cited first, 6 of 6)

- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [Multi-view Aggregation Network for Dichotomous Image Segmentation](https://mlsanity.com/paper/2404.07445/) (2024)
- [Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2308.05701/) (2023)
- [Surround-View Fisheye Optics in Computer Vision and Simulation: Survey and Challenges](https://mlsanity.com/paper/2402.12041/) (2024)
- [DENSER: 3D Gaussians Splatting for Scene Reconstruction of Dynamic Urban Environments](https://mlsanity.com/paper/2409.10041/) (2024)
- [LimSim Series: An Autonomous Driving Simulation Platform for Validation and Enhancement](https://mlsanity.com/paper/2502.09170/) (2025)

## Abstract (arXiv)

Simulation is an integral part in the process of developing autonomous vehicles and advantageous for training, validation, and verification of driving functions. Even though simulations come with a series of benefits compared to real-world experiments, various challenges still prevent virtual testing from entirely replacing physical test-drives. Our work provides an overview of these challenges with regard to different aspects and types of simulation and subsumes current trends to overcome them. We cover aspects around perception-, behavior- and content-realism as well as general hurdles in the domain of simulation. Among others, we observe a trend of data-driven, generative approaches and high-fidelity data synthesis to increasingly replace model-based simulation.