mlsanityData to 14 Oct 2026

Probing Multimodal LLMs as World Models for Driving

arXiv 2405.05956 ↗First posted 9 May 2024cs.RO cs.CVcited by 20

On the map: World models → World models for autonomous driving

also in 2 more
Abstract (arXiv)

We provide a sober look at the application of Multimodal Large Language Models (MLLMs) in autonomous driving, challenging common assumptions about their ability to interpret dynamic driving scenarios. Despite advances in models like GPT-4o, their performance in complex driving environments remains largely unexplored. Our experimental study assesses various MLLMs as world models using in-car camera perspectives and reveals that while these models excel at interpreting individual images, they struggle to synthesize coherent narratives across frames, leading to considerable inaccuracies in understanding (i) ego vehicle dynamics, (ii) interactions with other road actors, (iii) trajectory planning, and (iv) open-set scene reasoning. We introduce the Eval-LLM-Drive dataset and DriveSim simulator to enhance our evaluation, highlighting gaps in current MLLM capabilities and the need for improved models in dynamic real-world environments.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 46 of its 62 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

46 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
33 more
  1. 2022Language-driven Semantic Segmentationcited by 717
  2. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  3. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
  4. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  5. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  7. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  8. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  9. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  10. 2020Experience Grounds Languagecited by 242
  11. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  12. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  13. 2020Non-Local Spatial Propagation Network for Depth Completioncited by 195
  14. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 193
  15. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  16. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  17. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  18. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 111
  19. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  20. 2023Language Conditioned Traffic Generationcited by 87
  21. 2021Explanations in Autonomous Driving: A Surveycited by 81
  22. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  23. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
  24. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  25. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  26. 2023ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modelingcited by 60
  27. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  28. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarioscited by 46
  29. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  30. 2023Follow Anything: Open-set detection, tracking, and following in real-timecited by 29
  31. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  32. 2023ALT-Pilot: Autonomous navigation with Language augmented Topometric mapscited by 14
  33. 2021Trajectory Prediction with Linguistic Representationscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 6.

20 papers.

Sort:

By area:

  1. 2025A Survey of World Models for Autonomous Drivingcited by 60
  2. 2025FaceXBench: Evaluating Multimodal LLMs on Face Understandingcited by 24
  3. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  4. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  5. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  6. 2025V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Drivingcited by 14
  7. 2024Generating Out-Of-Distribution Scenarios Using Language Modelscited by 10
  8. 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
  9. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  10. 2025ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Modelscited by 9
  11. 2025RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Drivingcited by 8
  12. 2025PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systemscited by 4
8 more
  1. 2026See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selectioncited by 4
  2. 2025Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?cited by 2
  3. 2026Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Drivingcited by 2
  4. 2025Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samplescited by 1
  5. 2026SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videoscited by 0
  6. 2026CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Predictioncited by 0
  7. 2026Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fieldscited by 0
  8. 2026Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
  2. 2025A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  4. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  5. 2026GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving
  6. 2025Zero-Shot Scene Understanding with Multimodal Large Language Models for Automated Vehicles
  7. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  8. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving