# A Sim-to-Real Vision-based Lane Keeping System for a 1:10-scale Autonomous Vehicle

arXiv: https://arxiv.org/abs/2409.18097 · HTML: https://mlsanity.com/paper/2409.18097/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 9 of its 46 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)
- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 9 of 9)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys](https://mlsanity.com/paper/2303.17220/) (2023)
- [Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car](https://mlsanity.com/paper/1704.07911/) (2017)
- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [Robust Reinforcement Learning-based Autonomous Driving Agent for Simulation and Real World](https://mlsanity.com/paper/2009.11212/) (2020)
- [Sim-to-real reinforcement learning applied to end-to-end vehicle control](https://mlsanity.com/paper/2012.07461/) (2020)
- [Adaptive Lookahead Pure-Pursuit for Autonomous Racing](https://mlsanity.com/paper/2111.08873/) (2021)

## Abstract (arXiv)

In recent years, several competitions have highlighted the need to investigate vision-based solutions to address scenarios with functional insufficiencies in perception, world modeling and localization. This article presents the Vision-based Lane Keeping System (VbLKS) developed by the DEI-Unipd Team within the context of the Bosch Future Mobility Challenge 2022. The main contribution lies in a Simulation-to-Reality (Sim2Real) GPS-denied VbLKS for a 1:10-scale autonomous vehicle. In this VbLKS, the input to a tailored Pure Pursuit (PP) based control strategy, namely the Lookahead Heading Error (LHE), is estimated at a constant lookahead distance employing a Convolutional Neural Network (CNN). A training strategy for a compact CNN is proposed, emphasizing data generation and augmentation on simulated camera images from a 3D Gazebo simulator, and enabling real-time operation on low-level hardware. A tailored PP-based lateral controller equipped with a derivative action and a PP-based velocity reference generation are implemented. Tuning ranges are established through a systematic time-delay stability analysis. Validation in a representative controlled laboratory setting is provided.