mlsanityData to 13 Oct 2026

Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities

arXiv 2401.08045 ↗First posted 16 Jan 2024cs.CVcited by 24

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

The rise of large foundation models, trained on extensive datasets, is revolutionizing the field of AI. Models such as SAM, DALL-E2, and GPT-4 showcase their adaptability by extracting intricate patterns and performing effectively across diverse tasks, thereby serving as potent building blocks for a wide range of AI applications. Autonomous driving, a vibrant front in AI applications, remains challenged by the lack of dedicated vision foundation models (VFMs). The scarcity of comprehensive training data, the need for multi-sensor integration, and the diverse task-specific architectures pose significant obstacles to the development of VFMs in this field. This paper delves into the critical challenge of forging VFMs tailored specifically for autonomous driving, while also outlining future directions. Through a systematic analysis of over 250 papers, we dissect essential techniques for VFM development, including data preparation, pre-training strategies, and downstream task adaptation. Moreover, we explore key advancements such as NeRF, diffusion models, 3D Gaussian Splatting, and world models, presenting a comprehensive roadmap for future research. To empower researchers, we have built and maintained https://github.com/zhanghm1995/Forge_VFM4AD, an open-access repository constantly updated with the latest advancements in forging VFMs for autonomous driving.

Key facts. Cited by 24 papers in the map (data through 13 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 219 of its 258 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

219 of 258 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023End-to-end Autonomous Driving: Challenges and Frontiers29 Jun

Other references · 218

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,213
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  11. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,170
  12. 2022Generative Adversarial Networkscited by 5,614
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,840
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,516
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,498
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,435
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,240
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,440
  9. 2017CARLA: An Open Urban Driving Simulatorcited by 2,439
  10. 2022Video Diffusion Modelscited by 2,293
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,285
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,276
  13. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  14. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,958
  15. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,709
  16. 2020Knowledge Distillation: A Surveycited by 1,590
  17. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,581
  18. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,575
  19. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,467
  20. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,431
  21. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,334
  22. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  23. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,284
  24. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,269
  25. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,231
  26. 2023Mastering Diverse Domains through World Modelscited by 1,224
  27. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,215
  28. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,178
  29. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,150
  30. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  31. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,029
  32. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,023
  33. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  34. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  35. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 858
  36. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 847
  37. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  38. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763

50 most cited of 218 All 218 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 13; 2025: 9; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 341
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 20253D and 4D World Modeling: A Surveycited by 66
  5. 2025A Survey of World Models for Autonomous Drivingcited by 60
  6. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 53
  7. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  8. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  9. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  10. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  11. 2024VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Drivingcited by 15
  12. 2024UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representationscited by 10
12 more
  1. 2025LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarioscited by 7
  2. 2024Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Surveycited by 6
  3. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  4. 2025SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Drivingcited by 4
  5. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  6. 2024Uncertainty-Guided Enhancement on Driving Perception System via Foundation Modelscited by 3
  7. 2025PSA-SSL: Pose and Size-aware Self-Supervised Learning on LiDAR Point Cloudscited by 3
  8. 2026DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Drivingcited by 2
  9. 2024MOSS: Motion-based 3D Clothed Human Synthesis from Monocular Videocited by 1
  10. 2025Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trendscited by 1
  11. 2024BDC-Occ: Binarized Deep Convolution Unit For Binarized Occupancy Networkcited by 0
  12. 2026Training a Student Expert via Semi-Supervised Foundation Model Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey for Foundation Models in Autonomous Driving
  2. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
  3. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis
  4. 2026Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?
  5. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models
  6. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving
  7. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
  8. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook