# Pre-trained Transformer-Enabled Strategies with Human-Guided Fine-Tuning for End-to-end Navigation of Autonomous Vehicles arXiv: https://arxiv.org/abs/2402.12666 ยท HTML: https://mlsanity.com/paper/2402.12666/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. - Builds on 27 of its 43 references that are in the map. - First posted on arXiv 20 Feb 2024. ## Placed in - End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/) ## Builds on (most cited first, 27 of 27) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015) - [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018) - [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020) - [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021) - [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017) - [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022) - [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023) - [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021) - [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019) - [DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving](https://mlsanity.com/paper/2309.09777/) (2023) - [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022) - [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023) - [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021) - [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023) - [ReasonNet: End-to-End Driving with Temporal and Global Reasoning](https://mlsanity.com/paper/2305.10507/) (2023) - [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022) - [Socially Compliant Navigation through Raw Depth Inputs with Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1710.02543/) (2017) - [Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrations](https://mlsanity.com/paper/1805.07095/) (2018) - [Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2212.08787/) (2022) - [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021) - [Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving](https://mlsanity.com/paper/2109.12516/) (2021) - [Blending Anti-Aliasing into Vision Transformer](https://mlsanity.com/paper/2110.15156/) (2021) - [BEVGPT: Generative Pre-trained Large Model for Autonomous Driving Prediction, Decision-Making, and Planning](https://mlsanity.com/paper/2310.10357/) (2023) - [Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigation](https://mlsanity.com/paper/2301.00362/) (2023) - [Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments](https://mlsanity.com/paper/2207.01251/) (2022) ## Built on by (most cited first, 6 of 6) - [PP-TIL: Personalized Planning for Autonomous Driving with Instance-based Transfer Imitation Learning](https://mlsanity.com/paper/2407.18569/) (2024) - [NOLO: Navigate Only Look Once](https://mlsanity.com/paper/2408.01384/) (2024) - [Uncertainty-Guided Enhancement on Driving Perception System via Foundation Models](https://mlsanity.com/paper/2410.01144/) (2024) - [TrajMoE: Scene-Adaptive Trajectory Planning with Mixture of Experts and Reinforcement Learning](https://mlsanity.com/paper/2512.07135/) (2025) - [Game-Theoretic Risk-Shaped Reinforcement Learning for Safe Autonomous Driving](https://mlsanity.com/paper/2510.10960/) (2025) - [A Stitch in Time Saves Nine: Preserving Policy Compatibility Under Perception Updates in End-to-End Autonomous Driving](https://mlsanity.com/paper/2606.21509/) (2026) ## Abstract (arXiv) Autonomous driving (AD) technology, leveraging artificial intelligence, strives for vehicle automation. End-toend strategies, emerging to simplify traditional driving systems by integrating perception, decision-making, and control, offer new avenues for advanced driving functionalities. Despite their potential, current challenges include data efficiency, training complexities, and poor generalization. This study addresses these issues with a novel end-to-end AD training model, enhancing system adaptability and intelligence. The model incorporates a Transformer module into the policy network, undergoing initial behavior cloning (BC) pre-training for update gradients. Subsequently, fine-tuning through reinforcement learning with human guidance (RLHG) adapts the model to specific driving environments, aiming to surpass the performance limits of imitation learning (IL). The fine-tuning process involves human interactions, guiding the model to acquire more efficient and safer driving behaviors through supervision, intervention, demonstration, and reward feedback. Simulation results demonstrate that this framework accelerates learning, achieving precise control and significantly enhancing safety and reliability. Compared to other advanced baseline methods, the proposed approach excels in challenging AD tasks. The introduction of the Transformer module and human-guided fine-tuning provides valuable insights and methods for research and applications in the AD field.