# Lateral Ego-Vehicle Control without Supervision using Point Clouds

arXiv: https://arxiv.org/abs/2203.10662 · HTML: https://mlsanity.com/paper/2203.10662/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018)
- [Reinforcement Learning and Deep Learning based Lateral Control for Autonomous Driving](https://mlsanity.com/paper/1810.12778/) (2018)
- [End to End Vehicle Lateral Control Using a Single Fisheye Camera](https://mlsanity.com/paper/1808.06940/) (2018)
- [Towards Generalizing Sensorimotor Control Across Weather Conditions](https://mlsanity.com/paper/1907.11025/) (2019)
- [Self-Supervised Steering Angle Prediction for Vehicle Control Using Visual Odometry](https://mlsanity.com/paper/2103.11204/) (2021)

## Built on by (most cited first, 1 of 1)

- [LiDAR View Synthesis for Robust Vehicle Navigation Without Expert Labels](https://mlsanity.com/paper/2308.01424/) (2023)

## Abstract (arXiv)

Existing vision based supervised approaches to lateral vehicle control are capable of directly mapping RGB images to the appropriate steering commands. However, they are prone to suffering from inadequate robustness in real world scenarios due to a lack of failure cases in the training data. In this paper, a framework for training a more robust and scalable model for lateral vehicle control is proposed. The framework only requires an unlabeled sequence of RGB images. The trained model takes a point cloud as input and predicts the lateral offset to a subsequent frame from which the steering angle is inferred. The frame poses are in turn obtained from visual odometry. The point cloud is conceived by projecting dense depth maps into 3D. An arbitrary number of additional trajectories from this point cloud can be generated during training. This is to increase the robustness of the model. Online experiments show that the performance of our method is superior to that of the supervised model.