mlsanityData to 7 Oct 2026

SceneGen: Learning to Generate Realistic Traffic Scenes

arXiv 2101.06541 ↗First posted 16 Jan 2021cs.CV cs.AI cs.LGcited by 83

Not on the map. Why?

Abstract (arXiv)

We consider the problem of generating realistic traffic scenes automatically. Existing methods typically insert actors into the scene according to a set of hand-crafted heuristics and are limited in their ability to model the true complexity and diversity of real traffic scenes, thus inducing a content gap between synthesized traffic scenes versus real ones. As a result, existing simulators lack the fidelity necessary to train and test self-driving vehicles. To address this limitation, we present SceneGen, a neural autoregressive model of traffic scenes that eschews the need for rules and heuristics. In particular, given the ego-vehicle state and a high definition map of surrounding area, SceneGen inserts actors of various classes into the scene and synthesizes their sizes, orientations, and velocities. We demonstrate on two large-scale datasets SceneGen's ability to faithfully model distributions of real traffic scenes. Moreover, we show that SceneGen coupled with sensor simulation can be used to train perception models that generalize to the real world.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 31 of its 62 references that are in the map. First posted on arXiv 16 Jan 2021.

Builds on

31 of 62 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2018Group Normalizationcited by 1,514
  8. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  9. 2018Demystifying MMD GANscited by 1,268
  10. 2016Pixel Recurrent Neural Networkscited by 814
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  12. 2013Generating Sequences With Recurrent Neural Networkscited by 549
19 more
  1. 2018GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Modelscited by 406
  2. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  3. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  4. 2017A Neural Representation of Sketch Drawingscited by 311
  5. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  6. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  7. 2019Efficient Graph Generation with Graph Recurrent Attention Networkscited by 231
  8. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  9. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  10. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  11. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  12. 2018Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Datacited by 94
  13. 2020HDNET: Exploiting HD Maps for 3D Object Detectioncited by 78
  14. 2013A Deep and Tractable Density Estimatorcited by 67
  15. 2013RNADE: The real-valued neural autoregressive density-estimatorcited by 65
  16. 2020Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generationcited by 58
  17. 2018Deep Directional Statistics: Pose Estimation with Uncertainty Quantificationcited by 45
  18. 2018Augmented LiDAR Simulator for Autonomous Drivingcited by 41
  19. 2020Testing the Safety of Self-driving Vehicles by Simulating Perception and Predictioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 19; 2024: 20; 2025: 26; 2026: 7.

83 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  3. 2022V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Drivingcited by 157
  4. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  5. 2023The Waymo Open Sim Agents Challengecited by 86
  6. 2023Language Conditioned Traffic Generationcited by 86
  7. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  8. 2023Synthetic Datasets for Autonomous Driving: A Surveycited by 48
  9. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusioncited by 46
  10. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarioscited by 46
  11. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  12. 2024Generative Artificial Intelligence: A Systematic Review and Applicationscited by 37
38 more
  1. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  2. 2023Data-driven Traffic Simulation: A Comprehensive Reviewcited by 29
  3. 2022V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perceptioncited by 28
  4. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 27
  5. 2023DriveSceneGen: Generating Diverse and Realistic Driving Scenarios from Scratchcited by 26
  6. 2023CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulationcited by 26
  7. 2023Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Drivingcited by 26
  8. 2022Trust, but Verify: Cross-Modality Fusion for HD Map Change Detectioncited by 25
  9. 2023GINA-3D: Learning to Generate Implicit Neural Assets in the Wildcited by 25
  10. 2023SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Modelscited by 24
  11. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  12. 2021CausalAF: Causal Autoregressive Flow for Safety-Critical Driving Scenario Generationcited by 20
  13. 2024Promptable Closed-loop Traffic Simulationcited by 20
  14. 2025SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Modelcited by 20
  15. 2024Language-Driven Interactive Traffic Trajectory Generationcited by 19
  16. 2023Generating Driving Scenes with Diffusioncited by 16
  17. 2023Lift3D: Synthesize 3D Training Data by Lifting 2D GAN to 3D Generative Radiance Fieldcited by 15
  18. 2024DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstructioncited by 15
  19. 2024Extrapolated Urban View Synthesis Benchmarkcited by 13
  20. 2021Towards Optimal Strategies for Training Self-Driving Perception Models in Simulationcited by 12
  21. 2024Robust Fast Adaptation from Adversarially Explicit Task Distribution Generationcited by 12
  22. 2024SceneGenAgent: Precise Industrial Scene Generation with Coding Agentcited by 12
  23. 2025Generating Multimodal Driving Scenes via Next-Scene Predictioncited by 12
  24. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  25. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Modelscited by 11
  26. 2024UniGen: Unified Modeling of Initial Agent States and Trajectories for Generating Autonomous Driving Scenarioscited by 10
  27. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integrationcited by 9
  28. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  29. 2025Constrained Generative Modeling with Manually Bridged Diffusion Modelscited by 8
  30. 2025Continuous Locomotive Crowd Behavior Generationcited by 7
  31. 2025Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Modelscited by 7
  32. 2025Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Drivingcited by 7
  33. 2022Conditional Permutation Invariant Flowscited by 6
  34. 2023From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Drivingcited by 6
  35. 2023Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidancecited by 6
  36. 2024Reviving Life on the Edge: Joint Score-Based Graph Generation of Rich Edge Attributescited by 6
  37. 2024Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Surveycited by 6
  38. 2025AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Frameworkcited by 5

50 most cited of 83 All 83 (JSON)