mlsanityData to 7 Oct 2026

DriveGAN: Towards a Controllable High-Quality Neural Simulation

arXiv 2104.15060 ↗First posted 30 Apr 2021cs.CV cs.ROcited by 125

Not on the map. Why?

Cited by 47 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Realistic simulators are critical for training and verifying robotics systems. While most of the contemporary simulators are hand-crafted, a scaleable way to build simulators is to use machine learning to learn how the environment behaves in response to an action, directly from data. In this work, we aim to learn to simulate a dynamic environment directly in pixel-space, by watching unannotated sequences of frames and their associated action pairs. We introduce a novel high-quality neural simulator referred to as DriveGAN that achieves controllability by disentangling different components without supervision. In addition to steering controls, it also includes controls for sampling features of a scene, such as the weather as well as the location of non-player objects. Since DriveGAN is a fully differentiable simulator, it further allows for re-simulation of a given video sequence, offering an agent to drive through a recorded scene again, possibly taking different actions. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data. We showcase that our approach greatly surpasses the performance of previous data-driven simulators, and allows for new features not explored before.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 53 of its 67 references that are in the map. First posted on arXiv 30 Apr 2021.

Builds on

53 of 67 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  11. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  12. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
38 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  3. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  4. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  5. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  7. 2018Which Training Methods for GANs do actually Converge?cited by 672
  8. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  11. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  12. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  13. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  14. 2018Video-to-Video Synthesiscited by 388
  15. 2016Generating Videos with Scene Dynamicscited by 376
  16. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  17. 2016A Learned Representation For Artistic Stylecited by 340
  18. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  19. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  20. 2018Stochastic Video Generation with a Learned Priorcited by 244
  21. 2017Stochastic Variational Video Predictioncited by 234
  22. 2017Geometric GANcited by 232
  23. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  24. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  25. 2019Scaling Autoregressive Video Modelscited by 189
  26. 2018Stochastic Adversarial Video Predictioncited by 173
  27. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  28. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  29. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  30. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  31. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  32. 2016Video Pixel Networkscited by 108
  33. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  34. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  35. 2017Exploring the structure of a real-time, arbitrary neural artistic stylization networkcited by 89
  36. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  37. 2020World-Consistent Video-to-Video Synthesiscited by 69
  38. 2017Recurrent Environment Simulatorscited by 68

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 18; 2023: 26; 2024: 28; 2025: 28; 2026: 19.

125 papers.

Sort:

By area:

  1. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  5. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  6. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  7. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  8. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  9. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  10. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  11. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  12. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
38 more
  1. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  2. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  3. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  4. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  5. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  6. 2022Generating Long Videos of Dynamic Scenescited by 113
  7. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  8. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  9. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  10. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  11. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  12. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  13. 2025Controllable Video Generation: A Surveycited by 93
  14. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  15. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  16. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  17. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  18. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  19. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  20. 2025A Survey of World Models for Autonomous Drivingcited by 60
  21. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  22. 2022Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Imagecited by 55
  23. 2023OVTrack: Open-Vocabulary Multiple Object Trackingcited by 53
  24. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  25. 2023Persistent Nature: A Generative Model of Unbounded 3D Worldscited by 44
  26. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  27. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  28. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  29. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  30. 2022Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertioncited by 35
  31. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  32. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  33. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  34. 2022Playable Environments: Video Manipulation in Space and Timecited by 29
  35. 2024S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generationcited by 28
  36. 2024UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Drivingcited by 27
  37. 2025MaskGWM: A Generalizable Driving World Model with Video Mask Reconstructioncited by 27
  38. 2021Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Datacited by 26

50 most cited of 125 All 125 (JSON)