# Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Driving

arXiv: https://arxiv.org/abs/2406.13223 · HTML: https://mlsanity.com/paper/2406.13223/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 29 of 29)

- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [Model-free Deep Reinforcement Learning for Urban Autonomous Driving](https://mlsanity.com/paper/1904.09503/) (2019)
- [Hierarchical Imitation and Reinforcement Learning](https://mlsanity.com/paper/1803.00590/) (2018)
- [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022)
- [Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1902.00089/) (2019)
- [Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving](https://mlsanity.com/paper/2103.10690/) (2021)
- [Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships](https://mlsanity.com/paper/2011.04251/) (2020)
- [Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors](https://mlsanity.com/paper/2305.04412/) (2023)
- [Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function](https://mlsanity.com/paper/2103.01556/) (2021)
- [Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving](https://mlsanity.com/paper/2007.00178/) (2020)
- [A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning](https://mlsanity.com/paper/2101.06778/) (2021)
- [Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2011.04752/) (2020)
- [Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian](https://mlsanity.com/paper/2108.11623/) (2021)
- [DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networks](https://mlsanity.com/paper/2108.05030/) (2021)
- [Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarios](https://mlsanity.com/paper/2210.02300/) (2022)
- [Dynamically Conservative Self-Driving Planner for Long-Tail Cases](https://mlsanity.com/paper/2305.07497/) (2023)
- [Closing the Planning-Learning Loop with Application to Autonomous Driving](https://mlsanity.com/paper/2101.03834/) (2021)
- [Accelerating Reinforcement Learning for Autonomous Driving using Task-Agnostic and Ego-Centric Motion Skills](https://mlsanity.com/paper/2209.12072/) (2022)
- [Learning Interaction-aware Guidance Policies for Motion Planning in Dense Traffic Scenarios](https://mlsanity.com/paper/2107.04538/) (2021)
- [How to Learn from Risk: Explicit Risk-Utility Reinforcement Learning for Efficient and Safe Driving Strategies](https://mlsanity.com/paper/2203.08409/) (2022)

## Built on by (most cited first, 1 of 1)

- [Temporal Self-Imitation Learning](https://mlsanity.com/paper/2606.19752/) (2026)

## Abstract (arXiv)

Autonomous vehicles inevitably encounter a vast array of scenarios in real-world environments. Addressing long-tail scenarios, particularly those involving intensive interactions with numerous traffic participants, remains one of the most significant challenges in achieving high-level autonomous driving. Reinforcement learning (RL) offers a promising solution for such scenarios and allows autonomous vehicles to continuously self-evolve during interactions. However, traditional RL often requires trial and error from scratch in new scenarios, resulting in inefficient exploration of unknown states. Integrating RL with planning-based methods can significantly accelerate the learning process. Additionally, conventional RL methods lack robust safety mechanisms, making agents prone to collisions in dynamic environments in pursuit of short-term rewards. Many existing safe RL methods depend on environment modeling to identify reliable safety boundaries for constraining agent behavior. However, explicit environmental models can fail to capture the complexity of dynamic environments comprehensively. Inspired by the observation that human drivers rarely take risks in uncertain situations, this study introduces the concept of action timing and proposes a timing-aware RL method, In this approach, a "timing imagination" process previews the execution results of the agent's strategies at different time scales. The optimal execution timing is then projected to each decision moment, generating a dynamic safety factor to constrain actions. A planning-based method serves as a conservative baseline strategy in uncertain states. In two representative interaction scenarios, an unsignalized intersection and a roundabout, the proposed model outperforms the benchmark models in driving safety.