mlsanityData to 14 Oct 2026

SimNet: Learning Reactive Self-driving Simulations from Real-world Observations

arXiv 2105.12332 ↗First posted 26 May 2021cs.RO cs.CV cs.LGcited by 77

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

In this work, we present a simple end-to-end trainable machine learning system capable of realistically simulating driving experiences. This can be used for the verification of self-driving system performance without relying on expensive and time-consuming road testing. In particular, we frame the simulation problem as a Markov Process, leveraging deep neural networks to model both state distribution and transition function. These are trainable directly from the existing raw observations without the need for any handcrafting in the form of plant or kinematic models. All that is needed is a dataset of historical traffic episodes. Our formulation allows the system to construct never seen scenes that unfold realistically reacting to the self-driving car's behaviour. We train our system directly from 1,000 hours of driving logs and measure both realism, reactivity of the simulation as the two key properties of the simulation. At the same time, we apply the method to evaluate the performance of a recently proposed state-of-the-art ML planning system trained from human driving logs. We discover this planning system is prone to previously unreported causal confusion issues that are difficult to test by non-reactive simulation. To the best of our knowledge, this is the first work that directly merges highly realistic data-driven simulations with a closed-loop evaluation for self-driving vehicles. We make the data, code, and pre-trained models publicly available to further stimulate simulation development.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 22 of its 40 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

22 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021End-to-end Interpretable Neural Motion Planner17 Jan

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  4. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  5. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  6. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  7. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  8. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
  9. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 397
  10. 2020Learning Lane Graph Representations for Motion Forecastingcited by 382
  11. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 326
  12. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
9 more
  1. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  2. 2019Causal Confusion in Imitation Learningcited by 288
  3. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 240
  4. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 179
  5. 2019Multiple Futures Predictioncited by 178
  6. 2018Baidu Apollo EM Motion Plannercited by 100
  7. 2020TPNet: Trajectory Proposal Network for Motion Predictioncited by 80
  8. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  9. 2018Learning from Demonstration in the Wildcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 14; 2023: 15; 2024: 21; 2025: 16; 2026: 5.

77 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  2. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 242
  3. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 190
  4. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 139
  5. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  6. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
  7. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 96
  8. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  9. 2023The Waymo Open Sim Agents Challengecited by 86
  10. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  11. 2021SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policiescited by 69
  12. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
38 more
  1. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  2. 2024BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Predictioncited by 50
  3. 2024SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rolloutcited by 48
  4. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusioncited by 47
  5. 2022DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehiclescited by 37
  6. 2021Autonomy 2.0: Why is self-driving always 5 years away?cited by 30
  7. 2021PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulationcited by 29
  8. 2022Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Expertscited by 29
  9. 2023Rethinking Closed-loop Training for Autonomous Drivingcited by 29
  10. 2023Data-driven Traffic Simulation: A Comprehensive Reviewcited by 29
  11. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 28
  12. 2023DriveSceneGen: Generating Diverse and Realistic Driving Scenarios from Scratchcited by 27
  13. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 27
  14. 2024Improving Agent Behaviors with RL Fine-tuning for Autonomous Drivingcited by 27
  15. 2023Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023cited by 25
  16. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  17. 2023SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Modelscited by 24
  18. 2021DriverGym: Democratising Reinforcement Learning for Autonomous Drivingcited by 23
  19. 2022TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Drivingcited by 23
  20. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  21. 2024AdvDiffuser: Generating Adversarial Safety-Critical Driving Scenarios via Guided Diffusioncited by 20
  22. 2025SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Modelcited by 20
  23. 2022Critic Sequential Monte Carlocited by 16
  24. 2023Generating Driving Scenes with Diffusioncited by 16
  25. 2025UniMM: A Unified Mixture Model Framework for Multi-Agent Simulationcited by 13
  26. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  27. 2023Editing Driver Character: Socially-Controllable Behavior Generation for Interactive Traffic Simulationcited by 10
  28. 2024UniGen: Unified Modeling of Initial Agent States and Trajectories for Generating Autonomous Driving Scenarioscited by 10
  29. 2025LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarioscited by 10
  30. 2024LASIL: Learner-Aware Supervised Imitation Learning For Long-term Microscopic Traffic Simulationcited by 8
  31. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  32. 2022RITA: Boost Driving Simulators with Realistic Interactive Traffic Flowcited by 7
  33. 2024Human-AI Safety: A Descendant of Generative AI and Control Systems Safetycited by 7
  34. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Modelscited by 7
  35. 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happencited by 5
  36. 2025AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Frameworkcited by 5
  37. 2025DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learningcited by 5
  38. 2022Analyzing and Enhancing Closed-loop Stability in Reactive Simulationcited by 4

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Scaling Planning for Automated Driving using Simplistic Synthetic Data
  2. 2024Explainable deep learning improves human mental models of self-driving cars
  3. 2025Learning to Drive from a World Model
  4. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving
  5. 2026Learning-Based Behavior Planning for Automated Driving: Real-World Integration and Deployment
  6. 2021ObserveNet Control: A Vision-Dynamics Learning Approach to Predictive Control in Autonomous Vehicles
  7. 2022Discrete Control in Real-World Driving Environments using Deep Reinforcement Learning
  8. 2025RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies