mlsanityData to 7 Oct 2026

VaViM and VaVAM: Autonomous Driving through Video Generative Modeling

arXiv 2502.15672 ↗First posted 21 Feb 2025cs.CV cs.AI cs.ROcited by 34

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

We explore the potential of large-scale generative video models for autonomous driving, introducing an open-source auto-regressive video model (VaViM) and its companion video-action model (VaVAM) to investigate how video pre-training transfers to real-world driving. VaViM is a simple auto-regressive video model that predicts frames using spatio-temporal token sequences. We show that it captures the semantics and dynamics of driving scenes. VaVAM, the video-action model, leverages the learned representations of VaViM to generate driving trajectories through imitation learning. Together, the models form a complete perception-to-action pipeline. We evaluate our models in open- and closed-loop driving scenarios, revealing that video-based pre-training holds promise for autonomous driving. Key insights include the semantic richness of the learned representations, the benefits of scaling for video synthesis, and the complex relationship between model size, data, and safety metrics in closed-loop evaluations. We release code and model weights at https://github.com/valeoai/VideoActionModel

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 81 of its 99 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

81 of 99 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT27 Dec
  2. 2024InfinityDrive: Breaking Time Limits in Driving World Models2 Dec
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  4. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  5. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research12 Oct
  6. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  7. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  8. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2022Generative Adversarial Networkscited by 5,603
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  13. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  14. 2022Video Diffusion Modelscited by 2,280
  15. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  16. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  17. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  18. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  19. 2022Planning-oriented Autonomous Drivingcited by 1,068
  20. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  21. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  22. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  23. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  24. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  25. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  26. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  27. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  28. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  29. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  30. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  31. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  32. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  33. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  34. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  35. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  36. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  37. 2022MAGVIT: Masked Generative Video Transformercited by 341
  38. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 20.

34 papers.

Sort:

By area:

  1. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  2. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  3. 20253D and 4D World Modeling: A Surveycited by 64
  4. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  7. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 39
  8. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 37
  9. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  10. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  11. 2026Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Drivingcited by 26
  12. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 22
22 more
  1. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 19
  2. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
  3. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 12
  4. 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Drivingcited by 9
  5. 2026The DAWN of World-Action Interactive Modelscited by 7
  6. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
  7. 2025VFMF: World Modeling by Forecasting Vision Foundation Model Featurescited by 5
  8. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  9. 2026SimWAM: A Simple World Action Model for End-to-End Autonomous Drivingcited by 5
  10. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  11. 2026Discrete-WAM: Unified Discrete Vision-Action Token Editing for World-Policy Learningcited by 3
  12. 2025Are AI-Generated Driving Videos Ready for Autonomous Driving? A Diagnostic Evaluation Frameworkcited by 2
  13. 2026Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulationcited by 2
  14. 2026GeoWAM: Visual Geometry World Action Models for Autonomous Drivingcited by 2
  15. 2026Representations Before Pixels: Semantics-Guided Hierarchical Video Predictioncited by 1
  16. 2026BadDreamer: Transferable Backdoor Attacks against Video World Models for Autonomous Drivingcited by 1
  17. 2026ReWorld: Representation Learning for World Action Modelscited by 1
  18. 2026Adaptive-WAM: Quality-Guided Early-Exit Planning from Intermediate Video-Diffusion Featurescited by 1
  19. 2026World-Action Models for Robot Learning and Control: A Surveycited by 1
  20. 2026OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teachercited by 1
  21. 2026How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Modelscited by 0
  22. 2026Guiding End-to-End Driving Models with Endpoint-Constrained Trajectory Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  2. 2024DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving
  3. 2026Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving
  4. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
  5. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
  6. 2026FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving
  7. 2025ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos
  8. 2026FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory