mlsanityData to 14 Oct 2026

Vision-based DRL Autonomous Driving Agent with Sim2Real Transfer

arXiv 2305.11589 ↗First posted 19 May 2023cs.RO cs.LGcited by 2

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

To achieve fully autonomous driving, vehicles must be capable of continuously performing various driving tasks, including lane keeping and car following, both of which are fundamental and well-studied driving ones. However, previous studies have mainly focused on individual tasks, and car following tasks have typically relied on complete leader-follower information to attain optimal performance. To address this limitation, we propose a vision-based deep reinforcement learning (DRL) agent that can simultaneously perform lane keeping and car following maneuvers. To evaluate the performance of our DRL agent, we compare it with a baseline controller and use various performance metrics for quantitative analysis. Furthermore, we conduct a real-world evaluation to demonstrate the Sim2Real transfer capability of the trained DRL agent. To the best of our knowledge, our vision-based car following and lane keeping agent with Sim2Real transfer capability is the first of its kind.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 20 of its 40 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

20 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving14 Apr
  2. 2021Modified DDPG car-following model with a real-world human driving experience with CARLA simulator29 Dec

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  5. 2018Learning Dexterous In-Hand Manipulationcited by 687
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  8. 2017A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problemcited by 128
  9. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  10. 2018Driving Policy Transfer via Modularity and Abstractioncited by 92
  11. 2020A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye Viewcited by 72
  12. 2019Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Drivingcited by 59
6 more
  1. 2019Human-Like Autonomous Car-Following Model with Deep Reinforcement Learningcited by 48
  2. 2017A Deep Reinforcement Learning Chatbotcited by 36
  3. 2016End-to-End Deep Reinforcement Learning for Lane Keeping Assistcited by 27
  4. 2019Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfercited by 26
  5. 2022Sim-to-Real Domain Adaptation for Lane Detection and Classification in Autonomous Drivingcited by 20
  6. 2019Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learningcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Autonomous Driving Small-Scale Cars: A Survey of Recent Developmentcited by 6
  2. 2025Experimental investigation of pose informed reinforcement learning for skid-steered visual navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learning
  2. 2023Review of Deep Reinforcement Learning for Autonomous Driving
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Survey
  4. 2024Deep Reinforcement Learning in Autonomous Car Path Planning and Control: A Survey
  5. 2021WAD: A Deep Reinforcement Learning Agent for Urban Autonomous Driving
  6. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Driving
  7. 2025Evaluating Scenario-based Decision-making for Interactive Autonomous Driving Using Rational Criteria: A Survey
  8. 2023DRL-Based Trajectory Tracking for Motion-Related Modules in Autonomous Driving