# MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersections

arXiv: https://arxiv.org/abs/2307.16118 · HTML: https://mlsanity.com/paper/2307.16118/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 30 Jul 2023.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles](https://mlsanity.com/paper/2001.11231/) (2020)
- [Human-Like Decision Making for Autonomous Driving: A Noncooperative Game Theoretic Approach](https://mlsanity.com/paper/2005.11064/) (2020)
- [Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2203.07413/) (2022)
- [Multi-task Safe Reinforcement Learning for Navigating Intersections in Dense Traffic](https://mlsanity.com/paper/2202.09644/) (2022)
- [Graph Reinforcement Learning Application to Co-operative Decision-Making in Mixed Autonomy Traffic: Framework, Survey, and Challenges](https://mlsanity.com/paper/2211.03005/) (2022)

## Built on by (most cited first, 30 of 38)

- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2409.03272/) (2024)
- [A Survey on Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2506.24044/) (2025)
- [Distilling Multi-modal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2501.09757/) (2025)
- [AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learning](https://mlsanity.com/paper/2404.06345/) (2024)
- [Asynchronous Large Language Model Enhanced Planner for Autonomous Driving](https://mlsanity.com/paper/2406.14556/) (2024)
- [Visual Adversarial Attack on Vision-Language Models for Autonomous Driving](https://mlsanity.com/paper/2411.18275/) (2024)
- [Empowering Autonomous Driving with Large Language Models: A Safety Perspective](https://mlsanity.com/paper/2312.00812/) (2023)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art](https://mlsanity.com/paper/2403.16527/) (2024)
- [AD-H: Language-guided Autonomous Driving with Hierarchical Agents](https://mlsanity.com/paper/2406.03474/) (2024)
- [Applications of Large Scale Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2311.12144/) (2023)
- [Large Language Models for Human-like Autonomous Driving: A Survey](https://mlsanity.com/paper/2407.19280/) (2024)
- [TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning](https://mlsanity.com/paper/2502.01387/) (2025)
- [Towards Socially Responsive Autonomous Vehicles: A Reinforcement Learning Framework with Driving Priors and Coordination Awareness](https://mlsanity.com/paper/2309.09726/) (2023)
- [Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm](https://mlsanity.com/paper/2411.10869/) (2024)
- [DDM-Lag : A Diffusion-based Decision-making Model for Autonomous Vehicles with Lagrangian Safety Enhancement](https://mlsanity.com/paper/2401.03629/) (2024)
- [Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models](https://mlsanity.com/paper/2604.01259/) (2026)
- [Large Language Model guided Deep Reinforcement Learning for Decision Making in Autonomous Driving](https://mlsanity.com/paper/2412.18511/) (2024)
- [Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving](https://mlsanity.com/paper/2508.02028/) (2025)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [A Survey of Generative AI for Intelligent Transportation Systems: Road Transportation Perspective](https://mlsanity.com/paper/2312.08248/) (2023)
- [Leveraging Large Language Models (LLMs) for Traffic Management at Urban Intersections: The Case of Mixed Traffic Scenarios](https://mlsanity.com/paper/2408.00948/) (2024)
- [Generative Scenario Rollouts for End-to-End Autonomous Driving](https://mlsanity.com/paper/2601.11475/) (2026)
- [PADriver: Towards Personalized Autonomous Driving](https://mlsanity.com/paper/2505.05240/) (2025)
- [A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routing](https://mlsanity.com/paper/2509.04853/) (2025)
- [Option Discovery Using LLM-guided Semantic Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2503.19007/) (2025)

## Abstract (arXiv)

Autonomous driving technology is poised to transform transportation systems. However, achieving safe and accurate multi-task decision-making in complex scenarios, such as unsignalized intersections, remains a challenge for autonomous vehicles. This paper presents a novel approach to this issue with the development of a Multi-Task Decision-Making Generative Pre-trained Transformer (MTD-GPT) model. Leveraging the inherent strengths of reinforcement learning (RL) and the sophisticated sequence modeling capabilities of the Generative Pre-trained Transformer (GPT), the MTD-GPT model is designed to simultaneously manage multiple driving tasks, such as left turns, straight-ahead driving, and right turns at unsignalized intersections. We initially train a single-task RL expert model, sample expert data in the environment, and subsequently utilize a mixed multi-task dataset for offline GPT training. This approach abstracts the multi-task decision-making problem in autonomous driving as a sequence modeling task. The MTD-GPT model is trained and evaluated across several decision-making tasks, demonstrating performance that is either superior or comparable to that of state-of-the-art single-task decision-making models.