# Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part II: Control-Aware Radio Resource Allocation

arXiv: https://arxiv.org/abs/2311.11280 · HTML: https://mlsanity.com/paper/2311.11280/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 19 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges](https://mlsanity.com/paper/1907.09059/) (2019)
- [AoI-Aware Resource Allocation for Platoon-Based C-V2X Networks via Multi-Agent Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2105.04196/) (2021)
- [Age-of-Information vs. Value-of-Information Scheduling for Cellular Networked Control Systems](https://mlsanity.com/paper/1903.05356/) (2019)
- [Deep Reinforcement Learning Aided Platoon Control Relying on V2X Information](https://mlsanity.com/paper/2203.15781/) (2022)
- [Autonomous Platoon Control with Integrated Deep Reinforcement Learning and Dynamic Programming](https://mlsanity.com/paper/2206.07536/) (2022)
- [A Value of Information Framework for Latent Variable Models](https://mlsanity.com/paper/2008.08907/) (2020)

## Built on by (most cited first, 4 of 4)

- [An Adaptive CSI Feedback Model Based on BiLSTM for Massive MIMO-OFDM Systems](https://mlsanity.com/paper/2408.06359/) (2024)
- [VoI-Driven Joint Optimization of Control and Communication in Vehicular Digital Twin Network](https://mlsanity.com/paper/2505.07892/) (2025)
- [Learning Value of Information towards Joint Communication and Control in 6G V2X](https://mlsanity.com/paper/2505.06978/) (2025)
- [Multi-Agent Reinforcement Learning for V2X Resource Allocation: Disentangling MARL Challenges Through Benchmarking](https://mlsanity.com/paper/2603.06607/) (2026)

## Abstract (arXiv)

In Part I of this two-part paper (Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part I: Communication-Aware Vehicle Control), we decomposed the multi-timescale control and communications (MTCC) problem in Cellular Vehicle-to-Everything (C-V2X) system into a communication-aware Deep Reinforcement Learning (DRL)-based platoon control (PC) sub-problem and a control-aware DRL-based radio resource allocation (RRA) sub-problem. We focused on the PC sub-problem and proposed the MTCC-PC algorithm to learn an optimal PC policy given an RRA policy. In this paper (Part II), we first focus on the RRA sub-problem in MTCC assuming a PC policy is given, and propose the MTCC-RRA algorithm to learn the RRA policy. Specifically, we incorporate the PC advantage function in the RRA reward function, which quantifies the amount of PC performance degradation caused by observation delay. Moreover, we augment the state space of RRA with PC action history for a more well-informed RRA policy. In addition, we utilize reward shaping and reward backpropagation prioritized experience replay (RBPER) techniques to efficiently tackle the multi-agent and sparse reward problems, respectively. Finally, a sample- and computational-efficient training approach is proposed to jointly learn the PC and RRA policies in an iterative process. In order to verify the effectiveness of the proposed MTCC algorithm, we performed experiments using real driving data for the leading vehicle, where the performance of MTCC is compared with those of the baseline DRL algorithms.