mlsanityData to 7 Oct 2026

DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving

arXiv 2401.03641 ↗First posted 8 Jan 2024cs.RO cs.CVcited by 41

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

In the field of autonomous driving, two important features of autonomous driving car systems are the explainability of decision logic and the accuracy of environmental perception. This paper introduces DME-Driver, a new autonomous driving system that enhances the performance and reliability of autonomous driving system. DME-Driver utilizes a powerful vision language model as the decision-maker and a planning-oriented perception model as the control signal generator. To ensure explainable and reliable driving decisions, the logical decision-maker is constructed based on a large vision language model. This model follows the logic employed by experienced human drivers and makes decisions in a similar manner. On the other hand, the generation of accurate control signals relies on precise and detailed environmental perception, which is where 3D scene perception models excel. Therefore, a planning oriented perception model is employed as the signal generator. It translates the logical decisions made by the decision-maker into accurate control signals for the self-driving cars. To effectively train the proposed model, a new dataset for autonomous driving was created. This dataset encompasses a diverse range of human driver behaviors and their underlying motivations. By leveraging this dataset, our model achieves high-precision planning accuracy through a logical thinking process.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 42 of its 59 references that are in the map. First posted on arXiv 8 Jan 2024.

Builds on

42 of 59 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  11. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
30 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2022Planning-oriented Autonomous Drivingcited by 1,068
  4. 2016End to End Learning for Self-Driving Carscited by 1,000
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  7. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  9. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  10. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  11. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  12. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  13. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  14. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  15. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  16. 2018Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Carscited by 197
  17. 2019Self-Driving Cars: A Surveycited by 147
  18. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  19. 2022Differentiable Raycasting for Self-supervised Occupancy Forecastingcited by 83
  20. 2018Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detectioncited by 82
  21. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  22. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  23. 2023Semantic Anomaly Detection with Large Language Modelscited by 53
  24. 2015An Empirical Evaluation of Deep Learning on Highway Drivingcited by 35
  25. 2020Human-Like Decision Making for Autonomous Driving: A Noncooperative Game Theoretic Approachcited by 30
  26. 2016Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomycited by 27
  27. 2023VLPD: Context-Aware Pedestrian Detection via Vision-Language Semantic Self-Supervisioncited by 19
  28. 2023Language-Guided 3D Object Detection in Point Cloud for Autonomous Drivingcited by 19
  29. 2017Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cuescited by 9
  30. 2016All Weather Perception: Joint Data Association, Tracking, and Classification for Autonomous Ground Vehiclescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 20; 2026: 6.

41 papers.

Sort:

By area:

  1. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  2. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  3. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  4. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  5. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  6. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  7. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  8. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  9. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  10. 2024VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenescited by 22
  11. 2024AgentsCoMerge: Large Language Model Empowered Collaborative Decision Making for Ramp Mergingcited by 22
  12. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Drivingcited by 19
29 more
  1. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  2. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  3. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understandingcited by 13
  4. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  5. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  6. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  7. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  8. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 6
  9. 2024Edge-Cloud Collaborative Motion Planning for Autonomous Driving with Large Language Modelscited by 5
  10. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  11. 2025A Comprehensive LLM-powered Framework for Driving Intelligence Evaluationcited by 5
  12. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  13. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  14. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  15. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  16. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  17. 2026EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillationcited by 3
  18. 2024CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4Vcited by 2
  19. 2025SD++: Enhancing Standard Definition Maps by Incorporating Road Knowledge using LLMscited by 2
  20. 2025DriveMRP: Enhancing Vision-Language Models with Synthetic Motion Data for Motion Risk Predictioncited by 2
  21. 2025SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Drivingcited by 2
  22. 2024EraW-Net: Enhance-Refine-Align W-Net for Scene-Associated Driver Attention Estimationcited by 1
  23. 2025KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Modelscited by 1
  24. 2025All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehiclescited by 1
  25. 2026Grounding Driving VLA via Inverse Kinematicscited by 1
  26. 2026ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Drivingcited by 1
  27. 2025Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learningcited by 0
  28. 2025From Human Intention to Action Prediction: Intention-Driven End-to-End Autonomous Drivingcited by 0
  29. 2026PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ADriver-I: A General World Model for Autonomous Driving
  2. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving
  3. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  4. 2024Leveraging Large Language Models for Enhancing Autonomous Vehicle Perception
  5. 2024Hierarchical End-to-End Autonomous Driving: Integrating BEV Perception with Deep Reinforcement Learning
  6. 2025DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planning
  7. 2025LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinking
  8. 2025InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving