# Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach

arXiv: https://arxiv.org/abs/2312.07795 · HTML: https://mlsanity.com/paper/2312.07795/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 39 of its 74 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control](https://mlsanity.com/paper/1903.04527/) (2019)
- [UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning](https://mlsanity.com/paper/2110.07577/) (2021)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Policy Expansion for Bridging Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2302.00935/) (2023)
- [The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/2109.02869/) (2021)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [Transformers are Meta-Reinforcement Learners](https://mlsanity.com/paper/2206.06614/) (2022)
- [Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal control](https://mlsanity.com/paper/2112.10107/) (2021)
- [Efficient Pressure: Improving efficiency for signalized intersections](https://mlsanity.com/paper/2112.02336/) (2021)
- [Benchmarking Offline Reinforcement Learning on Real-Robot Hardware](https://mlsanity.com/paper/2307.15690/) (2023)
- [Learning an Interpretable Traffic Signal Control Policy](https://mlsanity.com/paper/1912.11023/) (2019)
- [Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2311.10747/) (2023)
- [InTAS -- The Ingolstadt Traffic Scenario for SUMO](https://mlsanity.com/paper/2011.11995/) (2020)

## Built on by (most cited first, 3 of 3)

- [DRDT3: Diffusion-Refined Decision Test-Time Training Model](https://mlsanity.com/paper/2501.06718/) (2025)
- [Spatiotemporal Decision Transformer for Traffic Coordination](https://mlsanity.com/paper/2602.02903/) (2026)
- [Emergency Preemption Without Online Exploration: A Decision Transformer Approach](https://mlsanity.com/paper/2603.22315/) (2026)

## Abstract (arXiv)

Efficient traffic signal control is critical for reducing traffic congestion and improving overall transportation efficiency. The dynamic nature of traffic flow has prompted researchers to explore Reinforcement Learning (RL) for traffic signal control (TSC). Compared with traditional methods, RL-based solutions have shown preferable performance. However, the application of RL-based traffic signal controllers in the real world is limited by the low sample efficiency and high computational requirements of these solutions. In this work, we propose DTLight, a simple yet powerful lightweight Decision Transformer-based TSC method that can learn policy from easily accessible offline datasets. DTLight novelly leverages knowledge distillation to learn a lightweight controller from a well-trained larger teacher model to reduce implementation computation. Additionally, it integrates adapter modules to mitigate the expenses associated with fine-tuning, which makes DTLight practical for online adaptation with minimal computation and only a few fine-tuning steps during real deployment. Moreover, DTLight is further enhanced to be more applicable to real-world TSC problems. Extensive experiments on synthetic and real-world scenarios show that DTLight pre-trained purely on offline datasets can outperform state-of-the-art online RL-based methods in most scenarios. Experiment results also show that online fine-tuning further improves the performance of DTLight by up to 42.6% over the best online RL baseline methods. In this work, we also introduce Datasets specifically designed for TSC with offline RL (referred to as DTRL). Our datasets and code are publicly available.