# VTGNet: A Vision-based Trajectory Generation Network for Autonomous Vehicles in Urban Environments

arXiv: https://arxiv.org/abs/2004.12591 · HTML: https://mlsanity.com/paper/2004.12591/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 13 of 13)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation](https://mlsanity.com/paper/2104.12861/) (2021)
- [Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning](https://mlsanity.com/paper/2107.08325/) (2021)
- [End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Driving](https://mlsanity.com/paper/2104.08862/) (2021)
- [DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networks](https://mlsanity.com/paper/2108.05030/) (2021)
- [Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning](https://mlsanity.com/paper/2109.08473/) (2021)
- [Imitation Learning of Hierarchical Driving Model: from Continuous Intention to Continuous Trajectory](https://mlsanity.com/paper/2010.10393/) (2020)
- [Reducing Overconfidence Predictions for Autonomous Driving Perception](https://mlsanity.com/paper/2202.07825/) (2022)
- [Domain Generalization for Vision-based Driving Trajectory Generation](https://mlsanity.com/paper/2109.13858/) (2021)
- [Autonomous Navigation through intersections with Graph ConvolutionalNetworks and Conditional Imitation Learning for Self-driving Cars](https://mlsanity.com/paper/2102.00675/) (2021)
- [ECAFormer: Low-light Image Enhancement using Cross Attention](https://mlsanity.com/paper/2406.13281/) (2024)
- [CARIL: Confidence-Aware Regression in Imitation Learning for Autonomous Driving](https://mlsanity.com/paper/2503.00783/) (2025)
- [Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving](https://mlsanity.com/paper/2605.25393/) (2026)

## Abstract (arXiv)

Traditional methods for autonomous driving are implemented with many building blocks from perception, planning and control, making them difficult to generalize to varied scenarios due to complex assumptions and interdependencies. Recently, the end-to-end driving method has emerged, which performs well and generalizes to new environments by directly learning from export-provided data. However, many existing methods on this topic neglect to check the confidence of the driving actions and the ability to recover from driving mistakes. In this paper, we develop an uncertainty-aware end-to-end trajectory generation method based on imitation learning. It can extract spatiotemporal features from the front-view camera images for scene understanding, and then generate collision-free trajectories several seconds into the future. The experimental results suggest that under various weather and lighting conditions, our network can reliably generate trajectories in different urban environments, such as turning at intersections and slowing down for collision avoidance. Furthermore, closed-loop driving tests suggest that the proposed method achieves better cross-scene/platform driving results than the state-of-the-art (SOTA) end-to-end control method, where our model can recover from off-center and off-orientation errors and capture 80% of dangerous cases with high uncertainty estimations.