# Generating and Evolving Reward Functions for Highway Driving with Large Language Models

arXiv: https://arxiv.org/abs/2406.10540 · HTML: https://mlsanity.com/paper/2406.10540/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Placed in

- End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/)

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [An Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/1811.12560/) (2018)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2309.10228/) (2023)
- [Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving](https://mlsanity.com/paper/2310.02251/) (2023)
- [Learning Navigation Behaviors End-to-End with AutoRL](https://mlsanity.com/paper/1809.10124/) (2018)
- [DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving](https://mlsanity.com/paper/2403.16996/) (2024)
- [Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1902.00089/) (2019)
- [Simulation-based reinforcement learning for real-world autonomous driving](https://mlsanity.com/paper/1911.12905/) (2019)
- [Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1901.00569/) (2019)
- [Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03118/) (2020)
- [TrafficGPT: Viewing, Processing and Interacting with Traffic Foundation Models](https://mlsanity.com/paper/2309.06719/) (2023)
- [LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs](https://mlsanity.com/paper/2312.04372/) (2023)
- [LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Driving](https://mlsanity.com/paper/2402.01246/) (2024)
- [Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous driving](https://mlsanity.com/paper/2309.05282/) (2023)
- [FollowNet: A Comprehensive Benchmark for Car-Following Behavior Modeling](https://mlsanity.com/paper/2306.05381/) (2023)
- [LangProp: A code optimization framework using Large Language Models applied to driving](https://mlsanity.com/paper/2401.10314/) (2024)
- [Engineering Safety Requirements for Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2403.16289/) (2024)
- [EnsembleFollower: A Hybrid Car-Following Framework Based On Reinforcement Learning and Hierarchical Planning](https://mlsanity.com/paper/2308.16008/) (2023)
- [Driving Style Encoder: Situational Reward Adaptation for General-Purpose Planning in Automated Driving](https://mlsanity.com/paper/1912.03509/) (2019)

## Built on by (most cited first, 8 of 8)

- [A Survey on Large Language Models for Automated Planning](https://mlsanity.com/paper/2502.12435/) (2025)
- [Evaluating Vision-Language Models as Evaluators in Path Planning](https://mlsanity.com/paper/2411.18711/) (2024)
- [From Words to Wheels: Automated Style-Customized Policy Generation for Autonomous Driving](https://mlsanity.com/paper/2409.11694/) (2024)
- [RoboMoRe: LLM-based Robot Co-design via Joint Optimization of Morphology and Reward](https://mlsanity.com/paper/2506.00276/) (2025)
- [Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategies](https://mlsanity.com/paper/2410.10212/) (2024)
- [A Large Language Model-Enhanced Q-learning for Capacitated Vehicle Routing Problem with Time Windows](https://mlsanity.com/paper/2505.06178/) (2025)
- [An Automated Reinforcement Learning Reward Design Framework with Large Language Model for Cooperative Platoon Coordination](https://mlsanity.com/paper/2504.19480/) (2025)
- [Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge](https://mlsanity.com/paper/2511.20726/) (2025)

## Abstract (arXiv)

Reinforcement Learning (RL) plays a crucial role in advancing autonomous driving technologies by maximizing reward functions to achieve the optimal policy. However, crafting these reward functions has been a complex, manual process in many practices. To reduce this complexity, we introduce a novel framework that integrates Large Language Models (LLMs) with RL to improve reward function design in autonomous driving. This framework utilizes the coding capabilities of LLMs, proven in other areas, to generate and evolve reward functions for highway scenarios. The framework starts with instructing LLMs to create an initial reward function code based on the driving environment and task descriptions. This code is then refined through iterative cycles involving RL training and LLMs' reflection, which benefits from their ability to review and improve the output. We have also developed a specific prompt template to improve LLMs' understanding of complex driving simulations, ensuring the generation of effective and error-free code. Our experiments in a highway driving simulator across three traffic configurations show that our method surpasses expert handcrafted reward functions, achieving a 22% higher average success rate. This not only indicates safer driving but also suggests significant gains in development productivity.