# Risk-anticipatory autonomous driving strategies considering vehicles' weights, based on hierarchical deep reinforcement learning

arXiv: https://arxiv.org/abs/2401.08661 · HTML: https://mlsanity.com/paper/2401.08661/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 21 of its 98 references that are in the map.
- First posted on arXiv 27 Dec 2023.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [A causal view of compositional zero-shot recognition](https://mlsanity.com/paper/2006.14610/) (2020)
- [Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1903.01344/) (2019)
- [Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1901.00569/) (2019)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning](https://mlsanity.com/paper/2002.02667/) (2020)
- [Data-driven Traffic Simulation: A Comprehensive Review](https://mlsanity.com/paper/2310.15975/) (2023)
- [Autonomous Highway Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.00035/) (2019)
- [Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control](https://mlsanity.com/paper/1910.12047/) (2019)
- [A Learning-based Discretionary Lane-Change Decision-Making Model with Driving Style Awareness](https://mlsanity.com/paper/2010.09533/) (2020)
- [Decision-making Strategy on Highway for Autonomous Vehicles using Deep Reinforcement Learning](https://mlsanity.com/paper/2007.08691/) (2020)
- [Predicting highway lane-changing maneuvers: A benchmark analysis of machine and ensemble learning algorithms](https://mlsanity.com/paper/2204.10807/) (2022)
- [Don't Get Yourself into Trouble! Risk-aware Decision-Making for Autonomous Vehicles](https://mlsanity.com/paper/2106.04625/) (2021)

## Built on by (most cited first, 4 of 4)

- [Effective Reinforcement Learning Control using Conservative Soft Actor-Critic](https://mlsanity.com/paper/2505.03356/) (2025)
- [Towards a Reward-Free Reinforcement Learning Framework for Vehicle Control](https://mlsanity.com/paper/2502.15262/) (2025)
- [Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving](https://mlsanity.com/paper/2506.23771/) (2025)
- [Orchestrate, Generate, Reflect: A VLM-Based Multi-Agent Collaboration Framework for Automated Driving Policy Learning](https://mlsanity.com/paper/2509.17042/) (2025)

## Abstract (arXiv)

Autonomous vehicles (AVs) have the potential to prevent accidents caused by drivers errors and reduce road traffic risks. Due to the nature of heavy vehicles, whose collisions cause more serious crashes, the weights of vehicles need to be considered when making driving strategies aimed at reducing the potential risks and their consequences in the context of autonomous driving. This study develops an autonomous driving strategy based on risk anticipation, considering the weights of surrounding vehicles and using hierarchical deep reinforcement learning. A risk indicator integrating surrounding vehicles weights, based on the risk field theory, is proposed and incorporated into autonomous driving decisions. A hybrid action space is designed to allow for left lane changes, right lane changes and car-following, which enables AVs to act more freely and realistically whenever possible. To solve the above hybrid decision-making problem, a hierarchical proximal policy optimization (HPPO) algorithm with an attention mechanism (AT-HPPO) is developed, providing great advantages in maintaining stable performance with high robustness and generalization. An indicator, potential collision energy in conflicts (PCEC), is newly proposed to evaluate the performance of the developed AV driving strategy from the perspective of the consequences of potential accidents. The performance evaluation results in simulation and dataset demonstrate that our model provides driving strategies that reduce both the likelihood and consequences of potential accidents, at the same time maintaining driving efficiency. The developed method is especially meaningful for AVs driving on highways, where heavy vehicles make up a high proportion of the traffic.