mlsanityData to 7 Oct 2026

Virtual to Real Reinforcement Learning for Autonomous Driving

arXiv 1704.03952 ↗First posted 13 Apr 2017cs.AI cs.CVcited by 79

Not on the map. Why?

Cited by 28 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning is considered as a promising direction for driving policy learning. However, training autonomous driving vehicle with reinforcement learning in real environment involves non-affordable trial-and-error. It is more desirable to first train in a virtual environment and then transfer to the real environment. In this paper, we propose a novel realistic translation network to make model trained in virtual environment be workable in real world. The proposed network can convert non-realistic virtual image input into a realistic one with similar scene structure. Given realistic frames as input, driving policy trained by reinforcement learning can nicely adapt to real world driving. Experiments show that our proposed virtual to real (VR) reinforcement learning (RL) works pretty well. To our knowledge, this is the first successful case of driving policy trained by reinforcement learning that can adapt to real world driving data.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2019: 1; 2020: 1; 2021: 26; 2022: 13; 2023: 21; 2024: 15; 2025: 2.

79 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  3. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  4. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  5. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  6. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  7. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  8. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  9. 2021Domain Adaptation In Reinforcement Learning Via Latent Unified State Representationcited by 33
  10. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  11. 2022Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learningcited by 27
  12. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
38 more
  1. 2023CFlowNets: Continuous Control with Generative Flow Networkscited by 19
  2. 2021A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learningcited by 18
  3. 2023On the Foundation of Distributionally Robust Reinforcement Learningcited by 18
  4. 2021Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Drivingcited by 17
  5. 2021Learning Interactive Driving Policies via Data-driven Simulationcited by 17
  6. 2024Human-compatible driving partners through data-regularized self-play reinforcement learningcited by 17
  7. 2024A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Drivingcited by 17
  8. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  9. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  10. 2023Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-artcited by 14
  11. 2022Learning Robotic Navigation from Experience: Principles, Methods, and Recent Resultscited by 13
  12. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  13. 2022Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarioscited by 12
  14. 2023Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritagecited by 12
  15. 2019Gaze Training by Modulated Dropout Improves Imitation Learningcited by 10
  16. 2021A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehiclescited by 10
  17. 2022Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twincited by 9
  18. 2024Quantum Advantage Actor-Critic for Reinforcement Learningcited by 9
  19. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  20. 2022Coordinate-Aligned Multi-Camera Collaboration for Active Multi-Object Trackingcited by 7
  21. 2022New Challenges in Reinforcement Learning: A Survey of Security and Privacycited by 7
  22. 2024Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutionscited by 7
  23. 2022SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Criticscited by 6
  24. 2023Reward Certification for Policy Smoothed Reinforcement Learningcited by 6
  25. 2024Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusioncited by 6
  26. 2021Instance-Aware Predictive Navigation in Multi-Agent Environmentscited by 5
  27. 2021Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Drivingcited by 5
  28. 2021A Versatile and Efficient Reinforcement Learning Framework for Autonomous Drivingcited by 5
  29. 2022Data generation using simulation technology to improve perception mechanism of autonomous vehiclescited by 5
  30. 2023Anytime-Constrained Reinforcement Learningcited by 5
  31. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
  32. 2023Online Prototype Alignment for Few-shot Policy Transfercited by 4
  33. 2023RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamicscited by 4
  34. 2023A Framework for Few-Shot Policy Transfer through Observation Mapping and Behavior Cloningcited by 4
  35. 2024CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecastingcited by 4
  36. 2021Goal-constrained Sparse Reinforcement Learning for End-to-End Drivingcited by 3
  37. 2021A Bayesian Approach to Reinforcement Learning of Vision-Based Vehicular Controlcited by 3
  38. 2021ARC: Adversarially Robust Control Policies for Autonomous Vehiclescited by 3

50 most cited of 79 All 79 (JSON)