# Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part I: Communication-Aware Vehicle Control

arXiv: https://arxiv.org/abs/2311.11281 · HTML: https://mlsanity.com/paper/2311.11281/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 70 references that are in the map.
- First posted on arXiv 19 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1902.00089/) (2019)
- [Reinforcement Learning with Random Delays](https://mlsanity.com/paper/2010.02966/) (2020)
- [Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges](https://mlsanity.com/paper/1907.09059/) (2019)
- [Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control](https://mlsanity.com/paper/1910.12047/) (2019)
- [Edge Intelligence for Autonomous Driving in 6G Wireless System: Design Challenges and Solutions](https://mlsanity.com/paper/2012.06992/) (2020)
- [Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays](https://mlsanity.com/paper/2108.07555/) (2021)
- [AoI-Aware Resource Allocation for Platoon-Based C-V2X Networks via Multi-Agent Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2105.04196/) (2021)
- [Hybrid Car-Following Strategy based on Deep Deterministic Policy Gradient and Cooperative Adaptive Cruise Control](https://mlsanity.com/paper/2103.03796/) (2021)
- [Deep Reinforcement Learning Aided Platoon Control Relying on V2X Information](https://mlsanity.com/paper/2203.15781/) (2022)
- [Autonomous Platoon Control with Integrated Deep Reinforcement Learning and Dynamic Programming](https://mlsanity.com/paper/2206.07536/) (2022)
- [A Secure Sensor Fusion Framework for Connected and Automated Vehicles under Sensor Attacks](https://mlsanity.com/paper/2103.00883/) (2021)
- [Deep reinforcement learning for scheduling in large-scale networked control systems](https://mlsanity.com/paper/1905.05992/) (2019)
- [Impact of Quantized Side Information on Subchannel Scheduling for Cellular V2X](https://mlsanity.com/paper/1807.04830/) (2018)
- [Efficient Power-Splitting and Resource Allocation for Cellular V2X Communications](https://mlsanity.com/paper/2007.06928/) (2020)
- [Task-oriented Scheduling for Networked Control Systems: An Age of Information-Aware Implementation on Software-defined Radios](https://mlsanity.com/paper/2202.09189/) (2022)

## Built on by (most cited first, 2 of 2)

- [An Adaptive CSI Feedback Model Based on BiLSTM for Massive MIMO-OFDM Systems](https://mlsanity.com/paper/2408.06359/) (2024)
- [Learning Value of Information towards Joint Communication and Control in 6G V2X](https://mlsanity.com/paper/2505.06978/) (2025)

## Abstract (arXiv)

An intelligent decision-making system enabled by Vehicle-to-Everything (V2X) communications is essential to achieve safe and efficient autonomous driving (AD), where two types of decisions have to be made at different timescales, i.e., vehicle control and radio resource allocation (RRA) decisions. The interplay between RRA and vehicle control necessitates their collaborative design. In this two-part paper (Part I and Part II), taking platoon control (PC) as an example use case, we propose a joint optimization framework of multi-timescale control and communications (MTCC) based on Deep Reinforcement Learning (DRL). In this paper (Part I), we first decompose the problem into a communication-aware DRL-based PC sub-problem and a control-aware DRL-based RRA sub-problem. Then, we focus on the PC sub-problem assuming an RRA policy is given, and propose the MTCC-PC algorithm to learn an efficient PC policy. To improve the PC performance under random observation delay, the PC state space is augmented with the observation delay and PC action history. Moreover, the reward function with respect to the augmented state is defined to construct an augmented state Markov Decision Process (MDP). It is proved that the optimal policy for the augmented state MDP is optimal for the original PC problem with observation delay. Different from most existing works on communication-aware control, the MTCC-PC algorithm is trained in a delayed environment generated by the fine-grained embedded simulation of C-V2X communications rather than by a simple stochastic delay model. Finally, experiments are performed to compare the performance of MTCC-PC with those of the baseline DRL algorithms.