mlsanityData to 14 Oct 2026

Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability

arXiv 2405.17398 ↗First posted 27 May 2024cs.CV cs.AIcited by 347

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen environments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista, a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients to address these limitations. To accurately predict real-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise an effective latent replacement approach to inject historical frames as priors for coherent long-horizon rollouts. For action controllability, we incorporate a versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through an efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments on multiple datasets show that Vista outperforms the most advanced general-purpose video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% in FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.

Key facts. Cited by 347 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 136 of its 149 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

136 of 149 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  2. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
  3. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)26 Feb
  4. 2024GenAD: Generative End-to-End Autonomous Driving18 Feb
  5. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Driving29 Dec
  6. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  7. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  8. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  9. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations20 Nov
  10. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  11. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  12. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb
3 more
  1. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct
  2. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar
  3. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun

Other references · 121

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
38 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2022Video Diffusion Modelscited by 2,305
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  8. 2023Consistency Modelscited by 1,822
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  13. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  14. 2023Mastering Diverse Domains through World Modelscited by 1,253
  15. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  16. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  17. 2022Planning-oriented Autonomous Drivingcited by 1,087
  18. 2020Mastering Atari with Discrete World Modelscited by 1,007
  19. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  20. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  21. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  22. 2024Genie: Generative Interactive Environmentscited by 747
  23. 2022On Distillation of Guided Diffusion Modelscited by 725
  24. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  25. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  26. 2023ModelScope Text-to-Video Technical Reportcited by 658
  27. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  28. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  29. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  30. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  31. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  32. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  33. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  34. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  35. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 485
  36. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  37. 2023Learning Interactive Real-World Simulatorscited by 458
  38. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455

50 most cited of 121 All 121 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 47; 2025: 150; 2026: 150.

347 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  2. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 252
  3. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  4. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 185
  5. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 174
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  7. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  8. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  9. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  10. 2025Pseudo-Simulation for Autonomous Drivingcited by 150
  11. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 140
  12. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
38 more
  1. 2025Unified Vision-Language-Action Modelcited by 134
  2. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  3. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 124
  4. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  5. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 117
  6. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  7. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 106
  8. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  9. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 100
  10. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  11. 2025Controllable Video Generation: A Surveycited by 98
  12. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  13. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 96
  14. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 95
  15. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 92
  16. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 90
  17. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  18. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 84
  19. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 78
  20. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 76
  21. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 69
  22. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  23. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 67
  24. 20253D and 4D World Modeling: A Surveycited by 67
  25. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  26. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 62
  27. 2025A Survey of World Models for Autonomous Drivingcited by 60
  28. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 56
  29. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  30. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  31. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  32. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 50
  33. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  34. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  35. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 47
  36. 2024DINO-Foresight: Looking into the Future with DINOcited by 45
  37. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 45
  38. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44

50 most cited of 347 All 347 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SimWAM: A Simple World Action Model for End-to-End Autonomous Driving
  2. 2026SV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous Driving
  3. 2026X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving
  4. 2026BranchDrive: A Branch-Structured Dataset for Action-Conditioned Driving Prediction
  5. 2025Vehicle Dynamics Embedded World Models for Autonomous Driving
  6. 2026EditWM: Event-Decomposed World Modeling with Incremental Correction for End-to-End Autonomous Driving
  7. 2021Learning to drive from a world on rails
  8. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving