mlsanityData to 14 Oct 2026

VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving

arXiv 2412.15544 ↗First posted 20 Dec 2024cs.RO cs.AI cs.CVcited by 51

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

In recent years, reinforcement learning (RL)-based methods for learning driving policies have gained increasing attention in the autonomous driving community and have achieved remarkable progress in various driving scenarios. However, traditional RL approaches rely on manually engineered rewards, which require extensive human effort and often lack generalizability. To address these limitations, we propose VLM-RL, a unified framework that integrates pre-trained Vision-Language Models (VLMs) with RL to generate reward signals using image observation and natural language goals. The core of VLM-RL is the contrasting language goal (CLG)-as-reward paradigm, which uses positive and negative language goals to generate semantic rewards. We further introduce a hierarchical reward synthesis approach that combines CLG-based semantic rewards with vehicle state information, improving reward stability and offering a more comprehensive reward signal. Additionally, a batch-processing technique is employed to optimize computational efficiency during training. Extensive experiments in the CARLA simulator demonstrate that VLM-RL outperforms state-of-the-art baselines, achieving a 10.5% reduction in collision rate, a 104.6% increase in route completion rate, and robust generalization to unseen driving scenarios. Furthermore, VLM-RL can seamlessly integrate almost any standard RL algorithms, potentially revolutionizing the existing RL paradigm that relies on manual reward engineering and enabling continuous performance improvements. The demo video and code can be accessed at: https://zilin-huang.github.io/VLM-RL-website.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 55 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

26 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models17 Aug
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  6. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  7. 2023Reward Design with Language Modelscited by 261
  8. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 207
  9. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
  10. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  11. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  12. 2023Vision-Language Models as a Source of Rewardscited by 66
11 more
  1. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 43
  2. 2024Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Drivingcited by 41
  3. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 39
  4. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  5. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  6. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 23
  7. 2024A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Drivingcited by 18
  8. 2024Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Modelscited by 15
  9. 2024HighwayLLM: Decision-Making and Navigation in Highway Driving with RL-Informed Language Modelcited by 9
  10. 2024Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Drivingcited by 8
  11. 2024LORD: Large Models based Opposite Reward Design for Autonomous Drivingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 26.

51 papers.

Sort:

By area:

  1. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  2. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  3. 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Modelscited by 30
  4. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  5. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  6. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  7. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 13
  8. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Modelscited by 12
  9. 2025Reflection-Based Task Adaptation for Self-Improving VLAcited by 8
  10. 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happencited by 5
  11. 2026HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Drivingcited by 5
  12. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
38 more
  1. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Processcited by 4
  2. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  3. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Drivingcited by 3
  4. 2025SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modelingcited by 3
  5. 2025Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidancecited by 3
  6. 2025Automated Reward Design for Gran Turismocited by 3
  7. 2026Training and Benchmarking Code Generation for Physics-Inspired Animationscited by 3
  8. 2025SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Drivingcited by 2
  9. 2025SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Drivingcited by 2
  10. 2025One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Modelscited by 1
  11. 2025COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learningcited by 1
  12. 2026Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Drivingcited by 1
  13. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
  14. 2026WCog-VLA: A Dual-Level World-Cognitive Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 1
  15. 2026Vision--Language Signals in Constrained RL: Safety Gains Without Anticipationcited by 1
  16. 2025ROAD: Responsibility-Oriented Reward Design for Reinforcement Learning in Autonomous Drivingcited by 0
  17. 2025STELAR-VISION: Self-Topology-Aware Efficient Learning for Aligned Reasoning in Visioncited by 0
  18. 2025Orchestrate, Generate, Reflect: A VLM-Based Multi-Agent Collaboration Framework for Automated Driving Policy Learningcited by 0
  19. 2025CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Drivingcited by 0
  20. 2025Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environmentscited by 0
  21. 2026Sparse Threats, Focused Defense: Criticality-Aware Robust Reinforcement Learning for Safe Autonomous Drivingcited by 0
  22. 2026ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Groundingcited by 0
  23. 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
  24. 2026MapVerse: A Benchmark for Geospatial Question Answering on Diverse Real-World Mapscited by 0
  25. 2026VidDoS: Universal Denial-of-Service Attack on Video-based Large Language Modelscited by 0
  26. 2026SignReasoner: Compositional Reasoning for Complex Traffic Sign Understanding via Functional Structure Unitscited by 0
  27. 2026MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learningcited by 0
  28. 2026Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Modelscited by 0
  29. 2026FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Drivingcited by 0
  30. 2026MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Drivingcited by 0
  31. 2026VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Drivingcited by 0
  32. 2026One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spottingcited by 0
  33. 2026Large Multimodal Agents for Intelligent Transportation Systems: Architectures, Evidence, and Deployment Challengescited by 0
  34. 2026SPVC: Structured and Panoptic Video Fixing for Cross-Dataset Driving Scene Renderingcited by 0
  35. 2026Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learningcited by 0
  36. 2026Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Modelscited by 0
  37. 2026Careful Judge: Safe and Efficient Human-AI Collaborative Decision Makingcited by 0
  38. 2026It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bankcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
  2. 2026DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving
  3. 2026Found-RL: foundation model-enhanced reinforcement learning for autonomous driving
  4. 2025IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
  5. 2025VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving
  6. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
  7. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning
  8. 2024Generating and Evolving Reward Functions for Highway Driving with Large Language Models