mlsanityData to 7 Oct 2026

Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving

arXiv 2410.22313 ↗First posted 29 Oct 2024cs.CV cs.ROcited by 156

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

End-to-end autonomous driving demonstrates strong planning capabilities with large-scale data but still struggles in complex, rare scenarios due to limited commonsense. In contrast, Large Vision-Language Models (LVLMs) excel in scene understanding and reasoning. The path forward lies in merging the strengths of both approaches. Previous methods using LVLMs to predict trajectories or control signals yield suboptimal results, as LVLMs are not well-suited for precise numerical predictions. This paper presents Senna, an autonomous driving system combining an LVLM (Senna-VLM) with an end-to-end model (Senna-E2E). Senna decouples high-level planning from low-level trajectory prediction. Senna-VLM generates planning decisions in natural language, while Senna-E2E predicts precise trajectories. Senna-VLM utilizes a multi-image encoding approach and multi-view prompts for efficient scene understanding. Besides, we introduce planning-oriented QAs alongside a three-stage training strategy, which enhances Senna-VLM's planning performance while preserving commonsense. Extensive experiments on two datasets show that Senna achieves state-of-the-art planning performance. Notably, with pre-training on a large-scale dataset DriveX and fine-tuning on nuScenes, Senna significantly reduces average planning error by 27.12% and collision rate by 33.33% over model without pre-training. We believe Senna's cross-scenario generalization and transferability are essential for achieving fully autonomous driving. Code and models will be released at https://github.com/hustvl/Senna.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 88 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

62 of 88 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  2. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2022Planning-oriented Autonomous Drivingcited by 1,068
  15. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  16. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  17. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  18. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  19. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  20. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  21. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  22. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  23. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  24. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  25. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  26. 2022MapTR: Structured Modeling and Learning for Online Vectorized HD Map Constructioncited by 311
  27. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  28. 2023Mathematical Capabilities of ChatGPTcited by 263
  29. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  30. 2016A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehiclescited by 251
  31. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  32. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  33. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  34. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  35. 2023MapTRv2: An End-to-End Framework for Online Vectorized HD Map Constructioncited by 192
  36. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  37. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  38. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 75; 2026: 79.

156 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  3. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  4. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  5. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  6. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  7. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  9. 2025DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Drivingcited by 64
  10. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  11. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 57
  12. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
38 more
  1. 2024AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Drivingcited by 35
  2. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  3. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  4. 2025IRL-VLA: Training an Vision-Language-Action Policy via Reward World Modelcited by 31
  5. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  6. 2025Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learningcited by 25
  7. 2025Black-Box Adversarial Attack on Vision Language Models for Autonomous Drivingcited by 23
  8. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  9. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Drivingcited by 21
  10. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusioncited by 20
  11. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  12. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  13. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  14. 2025DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planningcited by 17
  15. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  16. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 17
  17. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  18. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  19. 2025WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Drivingcited by 15
  20. 2026SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Modelcited by 15
  21. 2026DriveFuture: Future-Aware Latent World Models for Autonomous Drivingcited by 15
  22. 2025HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarioscited by 14
  23. 2024CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Modelcited by 12
  24. 2025FASIONAD++ : Integrating High-Level Instruction and Information Bottleneck in FAt-Slow fusION Systems for Enhanced Safety in Autonomous Driving with Adaptive Feedbackcited by 12
  25. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  26. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  27. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 12
  28. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  29. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  30. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  31. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 10
  32. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 10
  33. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  34. 2025MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Casescited by 9
  35. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  36. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  37. 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Drivingcited by 9
  38. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Drivingcited by 8

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning
  2. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
  3. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
  4. 2026NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving
  5. 2025Distilling Multi-modal Large Language Models for Autonomous Driving
  6. 2026PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving
  7. 2026LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving
  8. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning