mlsanityData to 7 Oct 2026

CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning

arXiv 2412.16201 ↗First posted 17 Dec 2024cs.RO cs.AI cs.LGcited by 5

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

also in 1 more
Abstract (arXiv)

This paper presents CLIP-RLDrive, a new reinforcement learning (RL)-based framework for improving the decision-making of autonomous vehicles (AVs) in complex urban driving scenarios, particularly in unsignalized intersections. To achieve this goal, the decisions for AVs are aligned with human-like preferences through Contrastive Language-Image Pretraining (CLIP)-based reward shaping. One of the primary difficulties in RL scheme is designing a suitable reward model, which can often be challenging to achieve manually due to the complexity of the interactions and the driving scenarios. To deal with this issue, this paper leverages Vision-Language Models (VLMs), particularly CLIP, to build an additional reward model based on visual and textual cues.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 24 of its 37 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

24 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Towards Knowledge-driven Autonomous Driving7 Dec

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2016OpenAI Gymcited by 1,684
11 more
  1. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  2. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  3. 2023Reward Design with Language Modelscited by 257
  4. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  5. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  6. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  7. 2023Training Language Models with Language Feedback at Scalecited by 125
  8. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  9. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  10. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  11. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
  2. 2025COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learningcited by 1
  3. 2025Decoupled Functional Evaluation of Autonomous Driving Models via Feature Map Quality Scoringcited by 0
  4. 2025D4C: Data-Free Quantization for Contrastive Language-Image Pre-training Modelscited by 0
  5. 2026Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving
  2. 2026DriveReward: A Comprehensive Dataset and Generative Vision-Language Reward Model for Autonomous Driving
  3. 2024Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Models
  4. 2026Found-RL: foundation model-enhanced reinforcement learning for autonomous driving
  5. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Driving
  6. 2024LORD: Large Models based Opposite Reward Design for Autonomous Driving
  7. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective
  8. 2025ROAD: Responsibility-Oriented Reward Design for Reinforcement Learning in Autonomous Driving