# Modified DDPG car-following model with a real-world human driving experience with CARLA simulator

arXiv: https://arxiv.org/abs/2112.14602 · HTML: https://mlsanity.com/paper/2112.14602/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 29 Dec 2021.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 20 of 20)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [How to Start Training: The Effect of Initialization and Architecture](https://mlsanity.com/paper/1803.01719/) (2018)
- [Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving](https://mlsanity.com/paper/2103.10690/) (2021)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Offline Reinforcement Learning with Pseudometric Learning](https://mlsanity.com/paper/2103.01948/) (2021)
- [End-to-End Deep Reinforcement Learning for Lane Keeping Assist](https://mlsanity.com/paper/1612.04340/) (2016)
- [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021)
- [Real-time Dynamic Object Detection for Autonomous Driving using Prior 3D-Maps](https://mlsanity.com/paper/1809.11036/) (2018)
- [Formulation and validation of a car-following model based on deep reinforcement learning](https://mlsanity.com/paper/2109.14268/) (2021)

## Built on by (most cited first, 9 of 9)

- [FollowNet: A Comprehensive Benchmark for Car-Following Behavior Modeling](https://mlsanity.com/paper/2306.05381/) (2023)
- [A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving](https://mlsanity.com/paper/2304.08235/) (2023)
- [Autonomous Driving Small-Scale Cars: A Survey of Recent Development](https://mlsanity.com/paper/2404.06229/) (2024)
- [Vessel-following model for inland waterways based on deep reinforcement learning](https://mlsanity.com/paper/2207.03257/) (2022)
- [Vision-based DRL Autonomous Driving Agent with Sim2Real Transfer](https://mlsanity.com/paper/2305.11589/) (2023)
- [Deep reinforcement learning-based longitudinal control strategy for automated vehicles at signalised intersections](https://mlsanity.com/paper/2505.08896/) (2025)
- [Ethics-Aware Safe Reinforcement Learning for Rare-Event Risk Control in Interactive Urban Driving](https://mlsanity.com/paper/2508.14926/) (2025)
- [Knowledge-data fusion dominated vehicle platoon dynamics modeling and analysis: A physics-encoded deep learning approach](https://mlsanity.com/paper/2502.08658/) (2025)
- [A Computationally Efficient Learning of Artificial Intelligence System Reliability Considering Error Propagation](https://mlsanity.com/paper/2603.18201/) (2026)

## Abstract (arXiv)

In the autonomous driving field, fusion of human knowledge into Deep Reinforcement Learning (DRL) is often based on the human demonstration recorded in a simulated environment. This limits the generalization and the feasibility of application in real-world traffic. We propose a two-stage DRL method to train a car-following agent, that modifies the policy by leveraging the real-world human driving experience and achieves performance superior to the pure DRL agent. Training a DRL agent is done within CARLA framework with Robot Operating System (ROS). For evaluation, we designed different driving scenarios to compare the proposed two-stage DRL car-following agent with other agents. After extracting the "good" behavior from the human driver, the agent becomes more efficient and reasonable, which makes this autonomous agent more suitable for Human-Robot Interaction (HRI) traffic.