# Reinforcement Learning and Deep Learning based Lateral Control for Autonomous Driving

arXiv: https://arxiv.org/abs/1810.12778 · HTML: https://mlsanity.com/paper/1810.12778/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2018.

## Built on by (most cited first, 18 of 18)

- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Driving](https://mlsanity.com/paper/2203.16792/) (2022)
- [Gym-ANM: Reinforcement Learning Environments for Active Network Management Tasks in Electricity Distribution Systems](https://mlsanity.com/paper/2103.07932/) (2021)
- [NashAE: Disentangling Representations through Adversarial Covariance Minimization](https://mlsanity.com/paper/2209.10677/) (2022)
- [A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles](https://mlsanity.com/paper/2105.14218/) (2021)
- [TC-Driver: Trajectory Conditioned Driving for Robust Autonomous Racing -- A Reinforcement Learning Approach](https://mlsanity.com/paper/2205.09370/) (2022)
- [Risk-sensitive Markov Decision Process and Learning under General Utility Functions](https://mlsanity.com/paper/2311.13589/) (2023)
- [A Reinforcement Learning Benchmark for Autonomous Driving in Intersection Scenarios](https://mlsanity.com/paper/2109.10557/) (2021)
- [Lane-Keeping Control of Autonomous Vehicles Through a Soft-Constrained Iterative LQR](https://mlsanity.com/paper/2311.16900/) (2023)
- [Efficient Perception, Planning, and Control Algorithm for Vision-Based Automated Vehicles](https://mlsanity.com/paper/2209.07042/) (2022)
- [Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2307.08033/) (2023)
- [Multi-task UNet architecture for end-to-end autonomous driving](https://mlsanity.com/paper/2112.08967/) (2021)
- [Lateral Ego-Vehicle Control without Supervision using Point Clouds](https://mlsanity.com/paper/2203.10662/) (2022)
- [Learning Autonomous Race Driving with Action Mapping Reinforcement Learning](https://mlsanity.com/paper/2406.14934/) (2024)
- [Multi-Stage Transmission Line Flow Control Using Centralized and Decentralized Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08430/) (2021)
- [Tuning Path Tracking Controllers for Autonomous Cars Using Reinforcement Learning](https://mlsanity.com/paper/2301.03363/) (2023)
- [Tube-Based Robust Control Strategy for Vision-Guided Autonomous Vehicles](https://mlsanity.com/paper/2503.18752/) (2025)
- [CycleRL: Sim-to-Real Deep Reinforcement Learning for Robust Autonomous Bicycle Control](https://mlsanity.com/paper/2603.15013/) (2026)

## Abstract (arXiv)

This paper investigates the vision-based autonomous driving with deep learning and reinforcement learning methods. Different from the end-to-end learning method, our method breaks the vision-based lateral control system down into a perception module and a control module. The perception module which is based on a multi-task learning neural network first takes a driver-view image as its input and predicts the track features. The control module which is based on reinforcement learning then makes a control decision based on these features. In order to improve the data efficiency, we propose visual TORCS (VTORCS), a deep reinforcement learning environment which is based on the open racing car simulator (TORCS). By means of the provided functions, one can train an agent with the input of an image or various physical sensor measurement, or evaluate the perception algorithm on this simulator. The trained reinforcement learning controller outperforms the linear quadratic regulator (LQR) controller and model predictive control (MPC) controller on different tracks. The experiments demonstrate that the perception module shows promising performance and the controller is capable of controlling the vehicle drive well along the track center with visual input.