# End-to-end Autonomous Driving Perception with Sequential Latent Representation Learning

arXiv: https://arxiv.org/abs/2003.12464 · HTML: https://mlsanity.com/paper/2003.12464/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2020.

## Built on by (most cited first, 11 of 11)

- [StyleDrive: Towards Driving-Style Aware Benchmarking of End-To-End Autonomous Driving](https://mlsanity.com/paper/2506.23982/) (2025)
- [MATRIX: Multi-Agent Trajectory Generation with Diverse Contexts](https://mlsanity.com/paper/2403.06041/) (2024)
- [Efficient Learning of Urban Driving Policies Using Bird's-Eye-View State Representations](https://mlsanity.com/paper/2305.19904/) (2023)
- [FocalAD: Local Motion Planning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.11419/) (2025)
- [Towards Learning Generalizable Driving Policies from Restricted Latent Representations](https://mlsanity.com/paper/2111.03688/) (2021)
- [Zero-Shot Cross-City Generalization in End-to-End Autonomous Driving: Self-Supervised versus Supervised Representations](https://mlsanity.com/paper/2603.11417/) (2026)
- [V2X-Lead: LiDAR-based End-to-End Autonomous Driving with Vehicle-to-Everything Communication Integration](https://mlsanity.com/paper/2309.15252/) (2023)
- [New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles](https://mlsanity.com/paper/2512.01882/) (2025)
- [History Encoding Representation Design for Human Intention Inference](https://mlsanity.com/paper/2106.02222/) (2021)
- [Certified Interpretability Robustness for Class Activation Mapping](https://mlsanity.com/paper/2301.11324/) (2023)
- [Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion](https://mlsanity.com/paper/2606.02979/) (2026)

## Abstract (arXiv)

Current autonomous driving systems are composed of a perception system and a decision system. Both of them are divided into multiple subsystems built up with lots of human heuristics. An end-to-end approach might clean up the system and avoid huge efforts of human engineering, as well as obtain better performance with increasing data and computation resources. Compared to the decision system, the perception system is more suitable to be designed in an end-to-end framework, since it does not require online driving exploration. In this paper, we propose a novel end-to-end approach for autonomous driving perception. A latent space is introduced to capture all relevant features useful for perception, which is learned through sequential latent representation learning. The learned end-to-end perception model is able to solve the detection, tracking, localization and mapping problems altogether with only minimum human engineering efforts and without storing any maps online. The proposed method is evaluated in a realistic urban driving simulator, with both camera image and lidar point cloud as sensor inputs. The codes and videos of this work are available at our github repo and project website.