mlsanityData to 14 Oct 2026

Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models

arXiv 2310.17642 ↗First posted 26 Oct 2023cs.RO cs.CV cs.LGcited by 59

On the map: End-to-end autonomous driving → End-to-end system architecture

Abstract (arXiv)

As autonomous driving technology matures, end-to-end methodologies have emerged as a leading strategy, promising seamless integration from perception to control via deep learning. However, existing systems grapple with challenges such as unexpected open set environments and the complexity of black-box models. At the same time, the evolution of deep learning introduces larger, multimodal foundational models, offering multi-modal visual and textual understanding. In this paper, we harness these multimodal foundation models to enhance the robustness and adaptability of autonomous driving systems, enabling out-of-distribution, end-to-end, multimodal, and more explainable autonomy. Specifically, we present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. To do so, we introduce a method to extract nuanced spatial (pixel/patch-aligned) features from transformers to enable the encapsulation of both spatial and semantic features. Our approach (i) demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations, and (ii) allows the incorporation of latent space simulation (via text) for improved training (data augmentation via text) and policy debugging. We encourage the reader to check our explainer video at https://www.youtube.com/watch?v=4n-DJf8vXxo&feature=youtu.be and to view the code and demos on our project webpage at https://drive-anywhere.github.io/.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. Builds on 45 of its 51 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

45 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Learning Stability Attention in Vision-based End-to-end Driving Policies5 Apr

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Segment Anythingcited by 8,234
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
32 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  4. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  5. 2016End to End Learning for Self-Driving Carscited by 1,009
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  7. 2022Language-driven Semantic Segmentationcited by 717
  8. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  9. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
  10. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  11. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  12. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  13. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  14. 2018Learning to Drive in a Daycited by 273
  15. 2020Experience Grounds Languagecited by 242
  16. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  17. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  18. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 149
  19. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 111
  20. 2018Driving Policy Transfer via Modularity and Abstractioncited by 92
  21. 2023Language Conditioned Traffic Generationcited by 87
  22. 2021Explanations in Autonomous Driving: A Surveycited by 81
  23. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  24. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  25. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  26. 2014Convolutional Feature Masking for Joint Object and Stuff Segmentationcited by 41
  27. 2018Variational End-to-End Navigation and Localizationcited by 34
  28. 2020Label Efficient Visual Abstractions for Autonomous Drivingcited by 33
  29. 2019Control Barrier Functions for Systems with High Relative Degreecited by 31
  30. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  31. 2023Follow Anything: Open-set detection, tracking, and following in real-timecited by 29
  32. 2021Trajectory Prediction with Linguistic Representationscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 22; 2025: 20; 2026: 13.

59 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  2. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  3. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 143
  4. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Drivingcited by 93
  5. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  6. 2024CarLLaVA: Vision language models for camera-only closed-loop drivingcited by 57
  7. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  8. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Modelscited by 47
  9. 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Drivingcited by 33
  10. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  11. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 27
  12. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Modelscited by 27
38 more
  1. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  2. 2025LangCoop: Collaborative Driving with Languagecited by 25
  3. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  4. 2024ROADWork: A Dataset and Benchmark for Learning to Recognize, Observe, Analyze and Drive Through Work Zonescited by 22
  5. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Drivingcited by 21
  6. 2024Probing Multimodal LLMs as World Models for Drivingcited by 20
  7. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 18
  8. 2025MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Drivingcited by 18
  9. 2024PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Drivingcited by 17
  10. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  11. 2024Large Language Models Powered Context-aware Motion Prediction in Autonomous Drivingcited by 13
  12. 2025CHAI: Command Hijacking against embodied AIcited by 11
  13. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 11
  14. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  15. 2024Generating Out-Of-Distribution Scenarios Using Language Modelscited by 10
  16. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  17. 2024Words in Motion: Extracting Interpretable Control Vectors for Motion Transformerscited by 9
  18. 2024A Survey on Human-AI Collaboration with Large Foundation Modelscited by 7
  19. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  20. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  21. 2024DualAD: Dual-Layer Planning for Reasoning in Autonomous Drivingcited by 6
  22. 2025DataS^3: Dataset Subset Selection for Specializationcited by 6
  23. 2024DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Drivingcited by 5
  24. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Processcited by 4
  25. 2026See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selectioncited by 4
  26. 2024Exploring Foundation Models in Remote Sensing Image Change Detection: A Comprehensive Surveycited by 3
  27. 2024Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Featurescited by 3
  28. 2026Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representationscited by 3
  29. 2025SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Drivingcited by 2
  30. 2026Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Drivingcited by 2
  31. 2026Efficient Test-Time Finetuning of LLMs via Convex Reconstruction and Gradient Cachingcited by 2
  32. 2026CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Drivingcited by 2
  33. 2024ABNet: Attention BarrierNet for Safe and Scalable Robot Learningcited by 1
  34. 2025CTSAC: Curriculum-Based Transformer Soft Actor-Critic for Goal-Oriented Robot Explorationcited by 1
  35. 2025SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Taskscited by 1
  36. 2025KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Modelscited by 1
  37. 2025Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samplescited by 1
  38. 2025Advancing Autonomous Driving System Testing: Demands, Challenges, and Future Directionscited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  3. 2024A Survey for Foundation Models in Autonomous Driving
  4. 2019Multimodal End-to-End Autonomous Driving
  5. 2023End-to-end Autonomous Driving using Deep Learning: A Systematic Review
  6. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities
  7. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Model
  8. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving