mlsanityData to 7 Oct 2026

RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving

arXiv 2412.07689 ↗First posted 10 Dec 2024cs.CV cs.MM cs.ROcited by 30

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Large Multimodal Models (LMMs) have demonstrated exceptional comprehension and interpretation capabilities in Autonomous Driving (AD) by incorporating large language models. Despite the advancements, current data-driven AD approaches tend to concentrate on a single dataset and specific tasks, neglecting their overall capabilities and ability to generalize. To bridge these gaps, we propose RoboTron-Drive, a general large multimodal model designed to process diverse data inputs, such as images and multi-view videos, while performing a broad spectrum of AD tasks, including perception, prediction, and planning. Initially, the model undergoes curriculum pre-training to process varied visual signals and perform basic visual comprehension and perception tasks. Subsequently, we augment and standardize various AD datasets to finetune the model, resulting in an all-in-one LMM for autonomous driving. To assess the general capabilities and generalization ability, we conduct evaluations on six public benchmarks and undertake zero-shot transfer on three unseen datasets, where RoboTron-Drive achieves state-of-the-art performance across all tasks. We hope RoboTron-Drive as a promising solution for AD in the real world. Project page with code: https://github.com/zhijian11/RoboTron-Drive.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 69 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

49 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases16 Apr
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  3. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models2 Jan
  4. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  5. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  6. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving6 Dec
  7. 2023Dolphins: Multimodal Language Model for Driving1 Dec
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 41

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
29 more
  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  3. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  5. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  6. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  7. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  8. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  9. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  10. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  11. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  12. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  13. 2023A Language Agent for Autonomous Drivingcited by 161
  14. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  15. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  16. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  17. 2022A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspectivecited by 100
  18. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  19. 2020Explainable Object-induced Action Decision for Autonomous Vehiclescited by 88
  20. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  21. 2024Embodied Understanding of Driving Scenarioscited by 63
  22. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  23. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  24. 2022Expansion and Shrinkage of Localization for Weakly-Supervised Semantic Segmentationcited by 37
  25. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulationcited by 20
  26. 2023BEVGPT: Generative Pre-trained Large Model for Autonomous Driving Prediction, Decision-Making, and Planningcited by 15
  27. 2023FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibrationcited by 13
  28. 2024AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Modelscited by 13
  29. 2023BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systemscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 14.

30 papers.

Sort:

By area:

  1. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  2. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  3. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  4. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
  5. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  6. 2025DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planningcited by 17
  7. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Drivingcited by 14
  8. 2025VDT-Auto: End-to-end Autonomous Driving with VLM-Guided Diffusion Transformerscited by 12
  9. 2025OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Criticcited by 11
  10. 2026AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Drivingcited by 9
  11. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  12. 2025Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Drivingcited by 4
18 more
  1. 2025MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machinescited by 3
  2. 2025iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoningcited by 3
  3. 2025DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Riskscited by 3
  4. 2025Traffic-MLLM: Curiosity-Regularized Supervised Learning for Traffic Scenario Case-Based Reasoningcited by 2
  5. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2
  6. 2026EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Drivingcited by 2
  7. 2026DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Drivingcited by 2
  8. 2025Progressive Bird's Eye View Perception for Safety-Critical Autonomous Driving: A Comprehensive Surveycited by 1
  9. 2026RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operationcited by 1
  10. 2026The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Modelscited by 1
  11. 2026XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environmentscited by 1
  12. 2025Are VLMs Ready for Lane Topology Awareness in Autonomous Driving?cited by 0
  13. 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?cited by 0
  14. 2026RoadTones: Tone Controllable Text Generation from Road Event Videoscited by 0
  15. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervisioncited by 0
  16. 2026OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMscited by 0
  17. 2026Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understandingcited by 0
  18. 2026GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ADriver-I: A General World Model for Autonomous Driving
  2. 2024Probing Multimodal LLMs as World Models for Driving
  3. 2024OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving
  4. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
  5. 2025DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  7. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding
  8. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs