# A Risk-Averse Preview-based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles

arXiv: https://arxiv.org/abs/2112.03232 · HTML: https://mlsanity.com/paper/2112.03232/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 26 of its 93 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 26 of 26)

- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Probabilistic Vehicle Trajectory Prediction over Occupancy Grid Map via Recurrent Neural Network](https://mlsanity.com/paper/1704.07049/) (2017)
- [Policy Gradients for CVaR-Constrained MDPs](https://mlsanity.com/paper/1405.2690/) (2014)
- [An Efficient Reachability-Based Framework for Provably Safe Autonomous Navigation in Unknown Environments](https://mlsanity.com/paper/1905.00532/) (2019)
- [A Risk-Sensitive Finite-Time Reachability Approach for Safety of Stochastic Dynamic Systems](https://mlsanity.com/paper/1902.11277/) (2019)
- [Risk-Averse Planning Under Uncertainty](https://mlsanity.com/paper/1909.12499/) (2019)
- [A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles](https://mlsanity.com/paper/2105.14218/) (2021)
- [Driver Modeling through Deep Reinforcement Learning and Behavioral Game Theory](https://mlsanity.com/paper/2003.11071/) (2020)
- [Safe Value Functions](https://mlsanity.com/paper/2105.12204/) (2021)
- [A Taxonomy of Similarity Metrics for Markov Decision Processes](https://mlsanity.com/paper/2103.04706/) (2021)
- [Data-driven Dynamic Multi-objective Optimal Control: An Aspiration-satisfying Reinforcement Learning Approach](https://mlsanity.com/paper/2005.07118/) (2020)
- [A Framework for Time-Consistent, Risk-Averse Model Predictive Control: Theory and Algorithms](https://mlsanity.com/paper/1511.06981/) (2015)
- [Risk-averse model predictive control](https://mlsanity.com/paper/1704.00342/) (2017)
- [Performance guarantees for model-based Approximate Dynamic Programming in continuous spaces](https://mlsanity.com/paper/1602.07273/) (2016)
- [Risk-Averse RRT* Planning with Nonlinear Steering and Tracking Controllers for Nonlinear Robotic Systems Under Uncertainty](https://mlsanity.com/paper/2103.05572/) (2021)
- [Assured Learning-enabled Autonomy: A Metacognitive Reinforcement Learning Framework](https://mlsanity.com/paper/2103.12558/) (2021)
- [Data-Driven Hierarchical Predictive Learning in Unknown Environments](https://mlsanity.com/paper/2005.05948/) (2020)
- [Online Learning Based Risk-Averse Stochastic MPC of Constrained Linear Uncertain Systems](https://mlsanity.com/paper/2011.11441/) (2020)
- [A Convex Programming Approach to Data-Driven Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2103.14606/) (2021)
- [Stochastic Optimal Growth Model with Risk Sensitive Preferences](https://mlsanity.com/paper/1509.05638/) (2015)
- [Safety Control with Preview Automaton](https://mlsanity.com/paper/1907.04935/) (2019)
- [Risk-sensitive safety specifications for stochastic systems using Conditional Value-at-Risk](https://mlsanity.com/paper/1909.09703/) (2019)
- [Long-run risk sensitive impulse control](https://mlsanity.com/paper/1912.02488/) (2019)
- [Learning-Based Risk-Averse Model Predictive Control for Adaptive Cruise Control with Stochastic Driver Models](https://mlsanity.com/paper/2005.02646/) (2020)

## Built on by (most cited first, 1 of 1)

- [Risk-Averse Total-Reward Reinforcement Learning](https://mlsanity.com/paper/2506.21683/) (2025)

## Abstract (arXiv)

A risk-averse preview-based Q-learning planner is presented for navigation of autonomous vehicles. To this end, the multi-lane road ahead of a vehicle is represented by a finite-state non-stationary Markov decision process (MDP). A risk assessment unit module is then presented that leverages the preview information provided by sensors along with a stochastic reachability module to assign reward values to the MDP states and update them as scenarios develop. A sampling-based risk-averse preview-based Q-learning algorithm is finally developed that generates samples using the preview information and reward function to learn risk-averse optimal planning strategies without actual interaction with the environment. The risk factor is imposed on the objective function to avoid fluctuation of the Q values, which can jeopardize the vehicle's safety and/or performance. The overall hybrid automaton model of the system is leveraged to develop a feasibility check unit module that detects unfeasible plans and enables the planner system to react proactively to the changes of the environment. Finally, to verify the efficiency of the presented algorithm, its implementation on two highway driving scenarios of an autonomous vehicle in a varying traffic density is considered.