# Distributional Soft Actor-Critic with Harmonic Gradient for Safe and Efficient Autonomous Driving in Multi-lane Scenarios

arXiv: https://arxiv.org/abs/2505.13532 · HTML: https://mlsanity.com/paper/2505.13532/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 13 of 13)

- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee](https://mlsanity.com/paper/2011.05869/) (2020)
- [Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors](https://mlsanity.com/paper/2001.02811/) (2020)
- [Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function](https://mlsanity.com/paper/2103.01556/) (2021)
- [Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligence](https://mlsanity.com/paper/2103.10290/) (2021)
- [Distributional Soft Actor-Critic with Three Refinements](https://mlsanity.com/paper/2310.05858/) (2023)
- [Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin](https://mlsanity.com/paper/2211.14874/) (2022)
- [Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data](https://mlsanity.com/paper/2001.09816/) (2020)
- [Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarios](https://mlsanity.com/paper/2109.05540/) (2021)
- [Self-learned Intelligence for Integrated Decision and Control of Automated Vehicles at Signalized Intersections](https://mlsanity.com/paper/2110.12359/) (2021)
- [Tackling Real-World Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/2207.02162/) (2022)
- [A Dynamic Penalty Function Approach for Constraints-Handling in Reinforcement Learning](https://mlsanity.com/paper/2012.11790/) (2020)

## Abstract (arXiv)

Reinforcement learning (RL), known for its self-evolution capability, offers a promising approach to training high-level autonomous driving systems. However, handling constraints remains a significant challenge for existing RL algorithms, particularly in real-world applications. In this paper, we propose a new safety-oriented training technique called harmonic policy iteration (HPI). At each RL iteration, it first calculates two policy gradients associated with efficient driving and safety constraints, respectively. Then, a harmonic gradient is derived for policy updating, minimizing conflicts between the two gradients and consequently enabling a more balanced and stable training process. Furthermore, we adopt the state-of-the-art DSAC algorithm as the backbone and integrate it with our HPI to develop a new safe RL algorithm, DSAC-H. Extensive simulations in multi-lane scenarios demonstrate that DSAC-H achieves efficient driving performance with near-zero safety constraint violations.