# How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence

arXiv: https://arxiv.org/abs/2305.01263 · HTML: https://mlsanity.com/paper/2305.01263/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 166 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 30 of 42)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data](https://mlsanity.com/paper/1909.00889/) (2019)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys](https://mlsanity.com/paper/2303.17220/) (2023)
- [Risk Averse Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1904.00511/) (2019)
- [Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning](https://mlsanity.com/paper/2103.14666/) (2021)
- [Meta Reinforcement Learning with Latent Variable Gaussian Processes](https://mlsanity.com/paper/1803.07551/) (2018)
- [SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic](https://mlsanity.com/paper/1911.04074/) (2019)
- [Curriculum Self-Paced Learning for Cross-Domain Object Detection](https://mlsanity.com/paper/1911.06849/) (2019)
- [Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World](https://mlsanity.com/paper/2003.07739/) (2020)

## Built on by (most cited first, 30 of 58)

- [DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation](https://mlsanity.com/paper/2607.05147/) (2026)
- [Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training](https://mlsanity.com/paper/2503.12030/) (2025)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [WcDT: World-centric Diffusion Transformer for Traffic Scene Generation](https://mlsanity.com/paper/2404.02082/) (2024)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Data-driven Traffic Simulation: A Comprehensive Review](https://mlsanity.com/paper/2310.15975/) (2023)
- [REvolve: Reward Evolution with Large Language Models using Human Feedback](https://mlsanity.com/paper/2406.01309/) (2024)
- [DriveVA: Video Action Models are Zero-Shot Drivers](https://mlsanity.com/paper/2604.04198/) (2026)
- [Causal Composition Diffusion Model for Closed-loop Traffic Generation](https://mlsanity.com/paper/2412.17920/) (2024)
- [Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap](https://mlsanity.com/paper/2403.16092/) (2024)
- [Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models](https://mlsanity.com/paper/2409.13661/) (2024)
- [Exploring Generative AI for Sim2Real in Driving Data Synthesis](https://mlsanity.com/paper/2404.09111/) (2024)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [TinyLidarNet: 2D LiDAR-based End-to-End Deep Learning Model for F1TENTH Autonomous Racing](https://mlsanity.com/paper/2410.07447/) (2024)
- [AI Simulation by Digital Twins: Systematic Survey, Reference Framework, and Mapping to a Standardized Architecture](https://mlsanity.com/paper/2506.06580/) (2025)
- [Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2403.18209/) (2024)
- [MetaSSC: Enhancing 3D Semantic Scene Completion for Autonomous Driving through Meta-Learning and Long-sequence Modeling](https://mlsanity.com/paper/2411.03672/) (2024)
- [SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model](https://mlsanity.com/paper/2503.13952/) (2025)
- [On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles](https://mlsanity.com/paper/2505.02274/) (2025)
- [Industrial-Grade Sensor Simulation via Gaussian Splatting: A Modular Framework for Scalable Editing and Full-Stack Validation](https://mlsanity.com/paper/2503.11731/) (2025)
- [DrivIng: A Large-Scale Multimodal Driving Dataset with Full Digital Twin Integration](https://mlsanity.com/paper/2601.15260/) (2026)
- [Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges](https://mlsanity.com/paper/2603.09086/) (2026)
- [Flexible Empowerment at Reasoning with Extended Best-of-N Sampling](https://mlsanity.com/paper/2604.15614/) (2026)
- [Learning Based NMPC Adaptation for Autonomous Driving using Parallelized Digital Twin](https://mlsanity.com/paper/2402.16645/) (2024)
- [A Twin Delayed Deep Deterministic Policy Gradient Algorithm for Autonomous Ground Vehicle Navigation via Digital Twin Perception Awareness](https://mlsanity.com/paper/2403.15067/) (2024)
- [Addressing Data Annotation Challenges in Multiple Sensors: A Solution for Scania Collected Datasets](https://mlsanity.com/paper/2403.18649/) (2024)
- [Edge Case Detection in Automated Driving: Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2410.08491/) (2024)
- [RALAD: Bridging the Real-to-Sim Domain Gap in Autonomous Driving with Retrieval-Augmented Learning](https://mlsanity.com/paper/2501.12296/) (2025)
- [Cutting-edge 3D reconstruction solutions for underwater coral reef images: A review and comparison](https://mlsanity.com/paper/2502.20154/) (2025)
- [RealMirror: A Comprehensive, Open-Source Vision-Language-Action Platform for Embodied AI](https://mlsanity.com/paper/2509.14687/) (2025)

## Abstract (arXiv)

Safety and cost are two important concerns for the development of autonomous driving technologies. From the academic research to commercial applications of autonomous driving vehicles, sufficient simulation and real world testing are required. In general, a large scale of testing in simulation environment is conducted and then the learned driving knowledge is transferred to the real world, so how to adapt driving knowledge learned in simulation to reality becomes a critical issue. However, the virtual simulation world differs from the real world in many aspects such as lighting, textures, vehicle dynamics, and agents' behaviors, etc., which makes it difficult to bridge the gap between the virtual and real worlds. This gap is commonly referred to as the reality gap (RG). In recent years, researchers have explored various approaches to address the reality gap issue, which can be broadly classified into three categories: transferring knowledge from simulation to reality (sim2real), learning in digital twins (DTs), and learning by parallel intelligence (PI) technologies. In this paper, we consider the solutions through the sim2real, DTs, and PI technologies, and review important applications and innovations in the field of autonomous driving. Meanwhile, we show the state-of-the-arts from the views of algorithms, models, and simulators, and elaborate the development process from sim2real to DTs and PI. The presentation also illustrates the far-reaching effects and challenges in the development of sim2real, DTs, and PI in autonomous driving.