# SPformer: A Transformer Based DRL Decision Making Method for Connected Automated Vehicles

arXiv: https://arxiv.org/abs/2409.15105 · HTML: https://mlsanity.com/paper/2409.15105/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [MapTRv2: An End-to-End Framework for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2308.05736/) (2023)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions](https://mlsanity.com/paper/2309.10150/) (2023)
- [Encoding word order in complex embeddings](https://mlsanity.com/paper/1912.12333/) (2019)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [Foundations of Sequence-to-Sequence Modeling for Time Series](https://mlsanity.com/paper/1805.03714/) (2018)
- [3DPPE: 3D Point Positional Encoding for Multi-Camera 3D Object Detection Transformers](https://mlsanity.com/paper/2211.14710/) (2022)
- [A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network](https://mlsanity.com/paper/2010.05437/) (2020)
- [Holistic Transformer: A Joint Neural Network for Trajectory Prediction and Decision-Making of Autonomous Vehicles](https://mlsanity.com/paper/2206.08809/) (2022)
- [Decision Making for Connected Automated Vehicles at Urban Intersections Considering Social and Individual Benefits](https://mlsanity.com/paper/2201.01428/) (2022)
- [ T3 OMVP: A Transformer-based Time and Team Reinforcement Learning Scheme for Observation-constrained Multi-Vehicle Pursuit in Urban Area](https://mlsanity.com/paper/2203.00183/) (2022)
- [Bringing Diversity to Autonomous Vehicles: An Interpretable Multi-vehicle Decision-making and Planning Framework](https://mlsanity.com/paper/2302.06803/) (2023)

## Built on by (most cited first, 1 of 1)

- [Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Quality](https://mlsanity.com/paper/2608.01133/) (2026)

## Abstract (arXiv)

In mixed autonomy traffic environment, every decision made by an autonomous-driving car may have a great impact on the transportation system. Because of the complex interaction between vehicles, it is challenging to make decisions that can ensure both high traffic efficiency and safety now and futher. Connected automated vehicles (CAVs) have great potential to improve the quality of decision-making in this continuous, highly dynamic and interactive environment because of their stronger sensing and communicating ability. For multi-vehicle collaborative decision-making algorithms based on deep reinforcement learning (DRL), we need to represent the interactions between vehicles to obtain interactive features. The representation in this aspect directly affects the learning efficiency and the quality of the learned policy. To this end, we propose a CAV decision-making architecture based on transformer and reinforcement learning algorithms. A learnable policy token is used as the learning medium of the multi-vehicle joint policy, the states of all vehicles in the area of interest can be adaptively noticed in order to extract interactive features among agents. We also design an intuitive physical positional encodings, the redundant location information of which optimizes the performance of the network. Simulations show that our model can make good use of all the state information of vehicles in traffic scenario, so as to obtain high-quality driving decisions that meet efficiency and safety objectives. The comparison shows that our method significantly improves existing DRL-based multi-vehicle cooperative decision-making algorithms.