mlsanityData to 7 Oct 2026

OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving

arXiv 2412.15208 ↗First posted 19 Dec 2024cs.CV cs.LG cs.ROcited by 93

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Since the advent of Multimodal Large Language Models (MLLMs), they have made a significant impact across a wide range of real-world applications, particularly in Autonomous Driving (AD). Their ability to process complex visual data and reason about intricate driving scenarios has paved the way for a new paradigm in end-to-end AD systems. However, the progress of developing end-to-end models for AD has been slow, as existing fine-tuning methods demand substantial resources, including extensive computational power, large-scale datasets, and significant funding. Drawing inspiration from recent advancements in inference computing, we propose OpenEMMA, an open-source end-to-end framework based on MLLMs. By incorporating the Chain-of-Thought reasoning process, OpenEMMA achieves significant improvements compared to the baseline when leveraging a diverse range of MLLMs. Furthermore, OpenEMMA demonstrates effectiveness, generalizability, and robustness across a variety of challenging driving scenarios, offering a more efficient and effective approach to autonomous driving. We release all the codes in https://github.com/taco-group/OpenEMMA.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 64 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

54 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  2. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model16 Feb
  3. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec

Other references · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  8. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  10. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  11. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  12. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  13. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  14. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  15. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  16. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  17. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  18. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  19. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  20. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  21. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  22. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  23. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  24. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  25. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  26. 2019Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Drivingcited by 137
  27. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 112
  28. 2022TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarioscited by 104
  29. 2022MonoDETR: Depth-guided Transformer for Monocular 3D Object Detectioncited by 85
  30. 2023EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimationcited by 79
  31. 2023ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modelingcited by 60
  32. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
  33. 2024Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Drivingcited by 35
  34. 2018YOLO3D: End-to-end real-time 3D Oriented Object Bounding Box Detection from LiDAR Point Cloudcited by 32
  35. 2022CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detectioncited by 30
  36. 2024CoMamba: Real-time Cooperative Perception Unlocked with State Space Modelscited by 9
  37. 2023Plum: Prompt Learning using Metaheuristiccited by 8
  38. 2023EqDrive: Efficient Equivariant Motion Forecasting with Multi-Modality for Autonomous Drivingcited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 49; 2026: 44.

93 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  3. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  4. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  5. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  6. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  7. 2025WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarioscited by 60
  8. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  9. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  10. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  11. 2025UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Drivingcited by 27
  12. 2025LangCoop: Collaborative Driving with Languagecited by 25
38 more
  1. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  2. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  3. 2025SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systemscited by 23
  4. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17
  5. 2025DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learningcited by 16
  6. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  7. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  8. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  9. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  10. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 15
  11. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoningcited by 14
  12. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 14
  13. 2025CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perceptioncited by 13
  14. 2025MindDrive: An All-in-One Framework Bridging World Models and Vision-Language Model for End-to-End Autonomous Drivingcited by 12
  15. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  16. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  17. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  18. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 10
  19. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  20. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  21. 2026AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Drivingcited by 9
  22. 2026CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planningcited by 9
  23. 2025Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarkscited by 8
  24. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  25. 2025ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Drivingcited by 7
  26. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Drivingcited by 6
  27. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  28. 2025Automated Vehicles Should be Connected with Natural Languagecited by 5
  29. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planningcited by 5
  30. 2026HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Drivingcited by 5
  31. 2026Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Modelscited by 5
  32. 2026Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Drivingcited by 5
  33. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Processcited by 4
  34. 2025Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Drivingcited by 4
  35. 2026NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Drivingcited by 4
  36. 2026Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planningcited by 4
  37. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Modelscited by 3
  38. 2025SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modelingcited by 3

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  3. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
  4. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving
  5. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding
  6. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  7. 2024Probing Multimodal LLMs as World Models for Driving
  8. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs