# Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving

arXiv: https://arxiv.org/abs/2109.12516 · HTML: https://mlsanity.com/paper/2109.12516/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [Making Efficient Use of Demonstrations to Solve Hard Exploration Problems](https://mlsanity.com/paper/1909.01387/) (2019)
- [Guided Exploration with Proximal Policy Optimization using a Single Demonstration](https://mlsanity.com/paper/2007.03328/) (2020)
- [Intervention Aided Reinforcement Learning for Safe and Practical Policy Optimization in Navigation](https://mlsanity.com/paper/1811.06187/) (2018)
- [Reinforcement Learning Control of Robotic Knee with Human in the Loop by Flexible Policy Iteration](https://mlsanity.com/paper/2006.09008/) (2020)

## Built on by (most cited first, 22 of 22)

- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2212.08787/) (2022)
- [Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving](https://mlsanity.com/paper/2409.00858/) (2024)
- [Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving](https://mlsanity.com/paper/2501.08096/) (2025)
- [Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving](https://mlsanity.com/paper/2302.03939/) (2023)
- [Pre-trained Transformer-Enabled Strategies with Human-Guided Fine-Tuning for End-to-end Navigation of Autonomous Vehicles](https://mlsanity.com/paper/2402.12666/) (2024)
- [Human-In-The-Loop Machine Learning for Safe and Ethical Autonomous Vehicles: Principles, Challenges, and Opportunities](https://mlsanity.com/paper/2408.12548/) (2024)
- [Safe Decision-making for Lane-change of Autonomous Vehicles via Human Demonstration-aided Reinforcement Learning](https://mlsanity.com/paper/2207.00448/) (2022)
- [LearningFlow: Automated Policy Learning Workflow for Urban Driving with Large Language Models](https://mlsanity.com/paper/2501.05057/) (2025)
- [Enhancing End-to-End Autonomous Driving Systems Through Synchronized Human Behavior Data](https://mlsanity.com/paper/2408.10908/) (2024)
- [Multi-Objective Reinforcement Learning for Adaptable Personalized Autonomous Driving](https://mlsanity.com/paper/2505.05223/) (2025)
- [Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Control](https://mlsanity.com/paper/2603.25968/) (2026)
- [Sampling Efficient Deep Reinforcement Learning through Preference-Guided Stochastic Exploration](https://mlsanity.com/paper/2206.09627/) (2022)
- [Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2307.08033/) (2023)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [AEGIS: Human Attention-based Explainable Guidance for Intelligent Vehicle Systems](https://mlsanity.com/paper/2504.05950/) (2025)
- [Multi-Agent Probabilistic Ensembles with Trajectory Sampling for Connected Autonomous Vehicles](https://mlsanity.com/paper/2312.13910/) (2023)
- [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025)
- [Multi-agent Uncertainty-Aware Pessimistic Model-Based Reinforcement Learning for Connected Autonomous Vehicles](https://mlsanity.com/paper/2503.20462/) (2025)
- [Bootstrapping Reinforcement Learning with Sub-optimal Policies for Autonomous Driving](https://mlsanity.com/paper/2509.04712/) (2025)
- [ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks](https://mlsanity.com/paper/2605.17458/) (2026)
- [ReF-HIL: Shaping the Critic around Human Action Neighborhoods for Efficient Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2609.37131/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) requires skillful definition and remarkable computational efforts to solve optimization and control problems, which could impair its prospect. Introducing human guidance into reinforcement learning is a promising way to improve learning performance. In this paper, a comprehensive human guidance-based reinforcement learning framework is established. A novel prioritized experience replay mechanism that adapts to human guidance in the reinforcement learning process is proposed to boost the efficiency and performance of the reinforcement learning algorithm. To relieve the heavy workload on human participants, a behavior model is established based on an incremental online learning method to mimic human actions. We design two challenging autonomous driving tasks for evaluating the proposed algorithm. Experiments are conducted to access the training and testing performance and learning mechanism of the proposed algorithm. Comparative results against the state-of-the-art methods suggest the advantages of our algorithm in terms of learning efficiency, performance, and robustness.