# Optimizing Trajectories for Highway Driving with Offline Reinforcement Learning

arXiv: https://arxiv.org/abs/2203.10949 · HTML: https://mlsanity.com/paper/2203.10949/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 18 of its 51 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 18 of 18)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [A Survey of Deep Learning Applications to Autonomous Vehicle Control](https://mlsanity.com/paper/1912.10773/) (2019)
- [Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving](https://mlsanity.com/paper/1905.02680/) (2019)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.10056/) (2018)
- [Deep Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1811.11329/) (2018)
- [Lane Change Decision-making through Deep Reinforcement Learning with Rule-based Constraints](https://mlsanity.com/paper/1904.00231/) (2019)
- [Autonomous Highway Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.00035/) (2019)
- [Deep Inverse Q-learning with Constraints](https://mlsanity.com/paper/2008.01712/) (2020)
- [Dynamic Interaction-Aware Scene Understanding for Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/1909.13582/) (2019)
- [Dynamic Input for Deep Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/1907.10994/) (2019)
- [Quadratic Q-network for Learning Continuous Control for Autonomous Vehicles](https://mlsanity.com/paper/1912.00074/) (2019)
- [Trajectory Optimization for Unknown Constrained Systems using Reinforcement Learning](https://mlsanity.com/paper/1903.05751/) (2019)
- [Combining Benefits from Trajectory Optimization and Deep Reinforcement Learning](https://mlsanity.com/paper/1910.09667/) (2019)
- [Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways](https://mlsanity.com/paper/2012.03234/) (2020)

## Built on by (most cited first, 4 of 4)

- [A Reinforcement Learning-Boosted Motion Planning Framework: Comprehensive Generalization Performance in Autonomous Driving](https://mlsanity.com/paper/2402.01465/) (2024)
- [Trajectory Planning Using Reinforcement Learning for Interactive Overtaking Maneuvers in Autonomous Racing Scenarios](https://mlsanity.com/paper/2404.10658/) (2024)
- [Learning to Sample: Reinforcement Learning-Guided Sampling for Autonomous Vehicle Motion Planning](https://mlsanity.com/paper/2509.24313/) (2025)
- [Reinforcement Learning-based Dynamic Adaptation for Sampling-Based Motion Planning in Agile Autonomous Driving](https://mlsanity.com/paper/2510.10567/) (2025)

## Abstract (arXiv)

Implementing an autonomous vehicle that is able to output feasible, smooth and efficient trajectories is a long-standing challenge. Several approaches have been considered, roughly falling under two categories: rule-based and learning-based approaches. The rule-based approaches, while guaranteeing safety and feasibility, fall short when it comes to long-term planning and generalization. The learning-based approaches are able to account for long-term planning and generalization to unseen situations, but may fail to achieve smoothness, safety and the feasibility which rule-based approaches ensure. Hence, combining the two approaches is an evident step towards yielding the best compromise out of both. We propose a Reinforcement Learning-based approach, which learns target trajectory parameters for fully autonomous driving on highways. The trained agent outputs continuous trajectory parameters based on which a feasible polynomial-based trajectory is generated and executed. We compare the performance of our agent against four other highway driving agents. The experiments are conducted in the Sumo simulator, taking into consideration various realistic, dynamically changing highway scenarios, including surrounding vehicles with different driver behaviors. We demonstrate that our offline trained agent, with randomly collected data, learns to drive smoothly, achieving velocities as close as possible to the desired velocity, while outperforming the other agents. Code, training data and details available at: https://nrgit.informatik.uni-freiburg. de/branka.mirchevska/offline-rl-tp.