# Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin

arXiv: https://arxiv.org/abs/2211.14874 · HTML: https://mlsanity.com/paper/2211.14874/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 27 Nov 2022.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 13 of 13)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Smooth Exploration for Robotic Reinforcement Learning](https://mlsanity.com/paper/2005.05719/) (2020)
- [AlphaStar: An Evolutionary Computation Perspective](https://mlsanity.com/paper/1902.01724/) (2019)
- [BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud](https://mlsanity.com/paper/2006.11436/) (2020)
- [Assessing Transferability from Simulation to Reality for Reinforcement Learning](https://mlsanity.com/paper/1907.04685/) (2019)
- [Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation](https://mlsanity.com/paper/2207.10821/) (2022)
- [Sim2real for Autonomous Vehicle Control using Executable Digital Twin](https://mlsanity.com/paper/2208.06226/) (2022)
- [Tackling Real-World Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/2207.02162/) (2022)

## Built on by (most cited first, 10 of 10)

- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [Knowledge Transfer from Simple to Complex: A Safe and Efficient Reinforcement Learning Framework for Autonomous Driving Decision-Making](https://mlsanity.com/paper/2410.14468/) (2024)
- [A Twin Delayed Deep Deterministic Policy Gradient Algorithm for Autonomous Ground Vehicle Navigation via Digital Twin Perception Awareness](https://mlsanity.com/paper/2403.15067/) (2024)
- [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025)
- [Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving](https://mlsanity.com/paper/2604.03497/) (2026)
- [Scalable heliostat surface predictions from focal spots: Sim-to-Real transfer of inverse Deep Learning Raytracing](https://mlsanity.com/paper/2504.03712/) (2025)
- [SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments](https://mlsanity.com/paper/2504.21454/) (2025)
- [Distributional Soft Actor-Critic with Harmonic Gradient for Safe and Efficient Autonomous Driving in Multi-lane Scenarios](https://mlsanity.com/paper/2505.13532/) (2025)
- [Adaptive Control in Autonomous Driving via Real-Time Recurrent RL](https://mlsanity.com/paper/2602.02236/) (2026)
- [From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving](https://mlsanity.com/paper/2603.17714/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) is a promising solution for autonomous vehicles to deal with complex and uncertain traffic environments. The RL training process is however expensive, unsafe, and time consuming. Algorithms are often developed first in simulation and then transferred to the real world, leading to a common sim2real challenge that performance decreases when the domain changes. In this paper, we propose a transfer learning process to minimize the gap by exploiting digital twin technology, relying on a systematic and simultaneous combination of virtual and real world data coming from vehicle dynamics and traffic scenarios. The model and testing environment are evolved from model, hardware to vehicle in the loop and proving ground testing stages, similar to standard development cycle in automotive industry. In particular, we also integrate other transfer learning techniques such as domain randomization and adaptation in each stage. The simulation and real data are gradually incorporated to accelerate and make the transfer learning process more robust. The proposed RL methodology is applied to develop a path following steering controller for an autonomous electric vehicle. After learning and deploying the real-time RL control policy on the vehicle, we obtained satisfactory and safe control performance already from the first deployment, demonstrating the advantages of the proposed digital twin based learning process.