# End to End Vehicle Lateral Control Using a Single Fisheye Camera

arXiv: https://arxiv.org/abs/1808.06940 · HTML: https://mlsanity.com/paper/1808.06940/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2018.

## Built on by (most cited first, 13 of 13)

- [GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving](https://mlsanity.com/paper/2111.08575/) (2021)
- [Learning Interactive Driving Policies via Data-driven Simulation](https://mlsanity.com/paper/2111.12137/) (2021)
- [Demonstration-Efficient Guided Policy Search via Imitation of Robust Tube MPC](https://mlsanity.com/paper/2109.09910/) (2021)
- [Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Driving](https://mlsanity.com/paper/2203.02401/) (2022)
- [Imitation Learning for End to End Vehicle Longitudinal Control with Forward Camera](https://mlsanity.com/paper/1812.05841/) (2018)
- [Convolution kernel adaptation to calibrated fisheye](https://mlsanity.com/paper/2402.01456/) (2024)
- [Disentangling and Vectorization: A 3D Visual Perception Approach for Autonomous Driving Based on Surround-View Fisheye Cameras](https://mlsanity.com/paper/2107.08862/) (2021)
- [Surround-view Fisheye BEV-Perception for Valet Parking: Dataset, Baseline and Distortion-insensitive Multi-task Framework](https://mlsanity.com/paper/2212.04111/) (2022)
- [Goal-constrained Sparse Reinforcement Learning for End-to-End Driving](https://mlsanity.com/paper/2103.09189/) (2021)
- [Evaluating Robustness over High Level Driving Instruction for Autonomous Driving](https://mlsanity.com/paper/2105.10014/) (2021)
- [Lateral Ego-Vehicle Control without Supervision using Point Clouds](https://mlsanity.com/paper/2203.10662/) (2022)
- [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025)
- [LineMarkNet: Line Landmark Detection for Valet Parking](https://mlsanity.com/paper/2309.10475/) (2023)

## Abstract (arXiv)

Convolutional neural networks are commonly used to control the steering angle for autonomous cars. Most of the time, multiple long range cameras are used to generate lateral failure cases. In this paper we present a novel model to generate this data and label augmentation using only one short range fisheye camera. We present our simulator and how it can be used as a consistent metric for lateral end-to-end control evaluation. Experiments are conducted on a custom dataset corresponding to more than 10000 km and 200 hours of open road driving. Finally we evaluate this model on real world driving scenarios, open road and a custom test track with challenging obstacle avoidance and sharp turns. In our simulator based on real-world videos, the final model was capable of more than 99% autonomy on urban road