mlsanityData to 7 Oct 2026

EMMA: End-to-End Multimodal Model for Autonomous Driving

arXiv 2410.23262 ↗First posted 30 Oct 2024cs.CV cs.AI cs.CLcited by 226

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built upon a multi-modal large language model foundation like Gemini, EMMA directly maps raw camera sensor data into various driving-specific outputs, including planner trajectories, perception objects, and road graph elements. EMMA maximizes the utility of world knowledge from the pre-trained large language models, by representing all non-sensor inputs (e.g. navigation instructions and ego vehicle status) and outputs (e.g. trajectories and 3D locations) as natural language text. This approach allows EMMA to jointly process various driving tasks in a unified language space, and generate the outputs for each task using task-specific prompts. Empirically, we demonstrate EMMA's effectiveness by achieving state-of-the-art performance in motion planning on nuScenes as well as competitive results on the Waymo Open Motion Dataset (WOMD). EMMA also yields competitive results for camera-primary 3D object detection on the Waymo Open Dataset (WOD). We show that co-training EMMA with planner trajectories, object detection, and road graph tasks yields improvements across all three domains, highlighting EMMA's potential as a generalist model for autonomous driving applications. We hope that our results will inspire research to further evolve the state of the art in autonomous driving model architectures.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 95 of its 106 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

95 of 106 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving25 Mar
  2. 2024VLP: Vision Language Planning for Autonomous Driving10 Jan

Other references · 93

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  9. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
  13. 2016End to End Learning for Self-Driving Carscited by 1,000
  14. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  15. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  16. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  18. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  19. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  20. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  23. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  24. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  25. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  26. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  27. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  28. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  29. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  30. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  31. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  32. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  33. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  34. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  35. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  36. 2019Learning by Cheatingcited by 361
  37. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  38. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 94; 2026: 124.

226 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  3. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  4. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  5. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  6. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  7. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Drivingcited by 93
  8. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  9. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  10. 2025WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarioscited by 60
  11. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  12. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
38 more
  1. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  2. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learningcited by 38
  3. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  4. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  5. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 34
  6. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
  7. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  8. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  9. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  10. 2025IRL-VLA: Training an Vision-Language-Action Policy via Reward World Modelcited by 31
  11. 2024RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivingcited by 30
  12. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  13. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  14. 2025BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Drivingcited by 26
  15. 2025CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Drivingcited by 25
  16. 2025LangCoop: Collaborative Driving with Languagecited by 25
  17. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  18. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  19. 2025FaceXBench: Evaluating Multimodal LLMs on Face Understandingcited by 23
  20. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  21. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Drivingcited by 21
  22. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  23. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  24. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 18
  25. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  26. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17
  27. 2025Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarioscited by 16
  28. 2025DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learningcited by 16
  29. 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervisioncited by 16
  30. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  31. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  32. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  33. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  34. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 15
  35. 2025Data Scaling Laws for End-to-End Autonomous Drivingcited by 14
  36. 2025HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarioscited by 14
  37. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Drivingcited by 14
  38. 2025Thinking in 360deg: Humanoid Visual Search in the Wildcited by 14

50 most cited of 226 All 226 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Probing Multimodal LLMs as World Models for Driving
  2. 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation
  3. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  5. 2024Embodied Understanding of Driving Scenarios
  6. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving
  7. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
  8. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding