# Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving

arXiv: https://arxiv.org/abs/2501.08096 · HTML: https://mlsanity.com/paper/2501.08096/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 25 of its 68 references that are in the map.
- First posted on arXiv 14 Jan 2025.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 25 of 25)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Count-Based Exploration with the Successor Representation](https://mlsanity.com/paper/1807.11622/) (2018)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings](https://mlsanity.com/paper/2008.06622/) (2020)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [Autonomous Highway Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.00035/) (2019)
- [Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving](https://mlsanity.com/paper/2109.12516/) (2021)
- [Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration](https://mlsanity.com/paper/2305.19476/) (2023)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimation](https://mlsanity.com/paper/2004.10439/) (2020)
- [Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States](https://mlsanity.com/paper/1809.03214/) (2018)
- [Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.15968/) (2023)
- [Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space](https://mlsanity.com/paper/2402.15790/) (2024)
- [Multi-agent Soft Actor-Critic Based Hybrid Motion Planner for Mobile Robots](https://mlsanity.com/paper/2112.06594/) (2021)
- [Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach](https://mlsanity.com/paper/2302.02179/) (2023)
- [Learning-Driven Exploration for Reinforcement Learning](https://mlsanity.com/paper/1906.06890/) (2019)

## Built on by (most cited first, 8 of 8)

- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [FeaXDrive: Feasibility-aware Trajectory-Centric Diffusion Planning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.12656/) (2026)
- [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025)
- [Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems](https://mlsanity.com/paper/2608.11592/) (2026)
- [Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving](https://mlsanity.com/paper/2506.23771/) (2025)
- [Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints](https://mlsanity.com/paper/2603.05842/) (2026)
- [Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving](https://mlsanity.com/paper/2603.20230/) (2026)
- [Comparison-Based Ordinal Learning for Proactive Driving Risk Assessment](https://mlsanity.com/paper/2607.11128/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has shown excellent performance in solving decision-making and control problems of autonomous driving, which is increasingly applied in diverse driving scenarios. However, driving is a multi-attribute problem, leading to challenges in achieving multi-objective compatibility for current RL methods, especially in both policy updating and policy execution. On the one hand, a single value evaluation network limits the policy updating in complex scenarios with coupled driving objectives. On the other hand, the common single-type action space structure limits driving flexibility or results in large behavior fluctuations during policy execution. To this end, we propose a Multi-objective Ensemble-Critic reinforcement learning method with Hybrid Parametrized Action for multi-objective compatible autonomous driving. Specifically, an advanced MORL architecture is constructed, in which the ensemble-critic focuses on different objectives through independent reward functions. The architecture integrates a hybrid parameterized action space structure, and the generated driving actions contain both abstract guidance that matches the hybrid road modality and concrete control commands. Additionally, an uncertainty-based exploration mechanism that supports hybrid actions is developed to learn multi-objective compatible policies more quickly. Experimental results demonstrate that, in both simulator-based and HighD dataset-based multi-lane highway scenarios, our method efficiently learns multi-objective compatible autonomous driving with respect to efficiency, action consistency, and safety.