mlsanityData to 14 Oct 2026

ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

arXiv 2503.19755 ↗First posted 25 Mar 2025cs.CVcited by 178

On the map: LLM reasoning → Chain-of-thought reasoning for driving

also in 1 more
Abstract (arXiv)

End-to-end (E2E) autonomous driving methods still struggle to make correct decisions in interactive closed-loop evaluation due to limited causal reasoning capability. Current methods attempt to leverage the powerful understanding and reasoning abilities of Vision-Language Models (VLMs) to resolve this dilemma. However, the problem is still open that few VLMs for E2E methods perform well in the closed-loop evaluation due to the gap between the semantic reasoning space and the purely numerical trajectory output in the action space. To tackle this issue, we propose ORION, a holistic E2E autonomous driving framework by vision-language instructed action generation. ORION uniquely combines a QT-Former to aggregate long-term history context, a Large Language Model (LLM) for driving scenario reasoning, and a generative planner for precision trajectory prediction. ORION further aligns the reasoning space and the action space to implement a unified E2E optimization for both visual question-answering (VQA) and planning tasks. Our method achieves an impressive closed-loop performance of 77.74 Driving Score (DS) and 54.62% Success Rate (SR) on the challenge Bench2Drive datasets, which outperforms state-of-the-art (SOTA) methods by a large margin of 14.28 DS and 19.61% SR.

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. Builds on 64 of its 76 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

64 of 76 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Generative Adversarial Networkscited by 5,627
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2022Planning-oriented Autonomous Drivingcited by 1,087
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  12. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  13. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  14. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  15. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  16. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  17. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  18. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
  19. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  20. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  21. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  22. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  23. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  24. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
  25. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  26. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  27. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 270
  28. 2023Mathematical Capabilities of ChatGPTcited by 268
  29. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 252
  30. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 248
  31. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  32. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  33. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  34. 2022A Contrastive Framework for Neural Text Generationcited by 225
  35. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  36. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 194
  37. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  38. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 169

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 51; 2026: 127.

178 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  2. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 198
  3. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 176
  4. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 134
  5. 2025DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planningcited by 94
  6. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 86
  7. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  8. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  9. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  10. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 58
  11. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  12. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
38 more
  1. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  2. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 38
  3. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  4. 2025IRL-VLA: Training an Vision-Language-Action Policy via Reward World Modelcited by 31
  5. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  6. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Drivingcited by 28
  7. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 25
  8. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  9. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  10. 2026DriveVA: Video Action Models are Zero-Shot Driverscited by 23
  11. 2025DVGT: Driving Visual Geometry Transformercited by 21
  12. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  13. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 20
  14. 2026Towards Generalizable Robotic Manipulation in Dynamic Environmentscited by 20
  15. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 19
  16. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 19
  17. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 19
  18. 2025BridgeDrive: Diffusion Bridge Policy for Closed-Loop Trajectory Planning in Autonomous Drivingcited by 18
  19. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  20. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 17
  21. 2025Latent Chain-of-Thought World Modeling for End-to-End Drivingcited by 15
  22. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  23. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 15
  24. 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 15
  25. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 13
  26. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  27. 2025MindDrive: An All-in-One Framework Bridging World Models and Vision-Language Model for End-to-End Autonomous Drivingcited by 12
  28. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 12
  29. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 12
  30. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  31. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Criticcited by 11
  32. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11
  33. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  34. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 11
  35. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  36. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  37. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 10
  38. 2026TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAMcited by 10

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
  2. 2024LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Driving
  3. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving
  4. 2025More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
  5. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
  6. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
  7. 2026Outcome-Guided Distillation: A Teacher-Student Framework to Advance VLM Reasoning in Autonomous Driving
  8. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process