mlsanityData to 7 Oct 2026

Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data

arXiv 2210.14721 ↗First posted 25 Oct 2022cs.LG cs.AIcited by 9

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

Abstract (arXiv)

Autonomous driving is complex, requiring sophisticated 3D scene understanding, localization, mapping, and control. Rather than explicitly modelling and fusing each of these components, we instead consider an end-to-end approach via reinforcement learning (RL). However, collecting exploration driving data in the real world is impractical and dangerous. While training in simulation and deploying visual sim-to-real techniques has worked well for robot manipulation, deploying beyond controlled workspace viewpoints remains a challenge. In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps, subsequently enabling real-world images to also be translated. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world. Our approach, which can be trained in 48 hours on 1 GPU, can perform equally as well as a classical perception and control stack that took thousands of engineering hours over several months to build. We hope this work motivates future end-to-end autonomous driving research.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 32 of its 52 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

32 of 52 references.

References · 32

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2015Return of Frustratingly Easy Domain Adaptationcited by 389
  11. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  12. 2017DualGAN: Unsupervised Dual Learning for Image-to-Image Translationcited by 310
20 more
  1. 2016Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networkscited by 307
  2. 2018Learning to Drive in a Daycited by 266
  3. 2018Unity: A General Platform for Intelligent Agentscited by 232
  4. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  5. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  6. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  7. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  8. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  9. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  10. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  11. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  12. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 65
  13. 2019Simulation-based reinforcement learning for real-world autonomous drivingcited by 51
  14. 2021NeBula: Quest for Robotic Autonomy in Challenging Environments; TEAM CoSTAR at the DARPA Subterranean Challengecited by 32
  15. 2021STEP: Stochastic Traversability Evaluation and Planning for Risk-Aware Off-road Navigationcited by 27
  16. 2018Fast Approximate Clearance Evaluation for Rovers with Articulated Suspension Systemscited by 6
  17. 2021Hybrid Imitative Planning with Geometric and Predictive Costs in Off-road Environmentscited by 5
  18. 2021Autonomous Off-road Navigation over Extreme Terrains with Perceptually-challenging Conditionscited by 4
  19. 2020Sim-To-Real Transfer for Miniature Autonomous Car Racingcited by 3
  20. 2022Bridging Sim2Real Gap Using Image Gradients for the Task of End-to-End Autonomous Drivingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 2.

9 papers.

  1. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  2. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  3. 2023Language-Conditioned Path Planningcited by 9
  4. 2024Sim-to-Real Grasp Detection with Global-to-Local RGB-D Adaptationcited by 4
  5. 2024Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractionscited by 4
  6. 2024Motion planning for off-road autonomous driving based on human-like cognition and weight adaptationcited by 1
  7. 2026MVAdapt: Zero-Shot Multi-Vehicle Adaptation for End-to-End Autonomous Drivingcited by 1
  8. 2025Vehicle Dynamics Embedded World Models for Autonomous Drivingcited by 0
  9. 2026MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving
  2. 2025Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
  3. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
  4. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
  5. 2026TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations
  6. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving
  7. 2026Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation
  8. 2026RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts