# When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Network

arXiv: https://arxiv.org/abs/2009.10601 · HTML: https://mlsanity.com/paper/2009.10601/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2020.

## Built on by (most cited first, 30 of 30)

- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoT](https://mlsanity.com/paper/2202.01512/) (2022)
- [Machine Learning for Large-Scale Optimization in 6G Wireless Networks](https://mlsanity.com/paper/2301.03377/) (2023)
- [Resource-constrained Federated Edge Learning with Heterogeneous Data: Formulation and Analysis](https://mlsanity.com/paper/2110.07567/) (2021)
- [Cooperative Multi-Agent Reinforcement Learning: Asynchronous Communication and Linear Function Approximation](https://mlsanity.com/paper/2305.06446/) (2023)
- [SemiPFL: Personalized Semi-Supervised Federated Learning Framework for Edge Intelligence](https://mlsanity.com/paper/2203.08176/) (2022)
- [New Challenges in Reinforcement Learning: A Survey of Security and Privacy](https://mlsanity.com/paper/2301.00188/) (2022)
- [FedHQL: Federated Heterogeneous Q-Learning](https://mlsanity.com/paper/2301.11135/) (2023)
- [Applications of Explainable AI for 6G: Technical Aspects, Use Cases, and Research Challenges](https://mlsanity.com/paper/2112.04698/) (2021)
- [FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data](https://mlsanity.com/paper/2512.02350/) (2025)
- [ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling](https://mlsanity.com/paper/2210.02119/) (2022)
- [Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey](https://mlsanity.com/paper/2108.04087/) (2021)
- [Collaborative Policy Learning for Dynamic Scheduling Tasks in Cloud-Edge-Terminal IoT Networks Using Federated Reinforcement Learning](https://mlsanity.com/paper/2307.00541/) (2023)
- [Federated Learning: A Cutting-Edge Survey of the Latest Advancements and Applications](https://mlsanity.com/paper/2310.05269/) (2023)
- [Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving](https://mlsanity.com/paper/2402.05146/) (2024)
- [Intelligent Task Offloading: Advanced MEC Task Offloading and Resource Management in 5G Networks](https://mlsanity.com/paper/2501.06242/) (2025)
- [Federated Ensemble Model-based Reinforcement Learning in Edge Computing](https://mlsanity.com/paper/2109.05549/) (2021)
- [Federated Prompt-based Decision Transformer for Customized VR Services in Mobile Edge Computing System](https://mlsanity.com/paper/2402.09729/) (2024)
- [FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architectures](https://mlsanity.com/paper/2505.06497/) (2025)
- [Computation Offloading and Resource Allocation in F-RANs: A Federated Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2206.05881/) (2022)
- [Privacy-Preserving Joint Edge Association and Power Optimization for the Internet of Vehicles via Federated Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2301.11014/) (2023)
- [User-centric Heterogeneous-action Deep Reinforcement Learning for Virtual Reality in the Metaverse over Wireless Networks](https://mlsanity.com/paper/2302.01471/) (2023)
- [PeersimGym: An Environment for Solving the Task Offloading Problem with Reinforcement Learning](https://mlsanity.com/paper/2403.17637/) (2024)
- [Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency](https://mlsanity.com/paper/2405.17471/) (2024)
- [Federated Offline Policy Optimization with Dual Regularization](https://mlsanity.com/paper/2405.17474/) (2024)
- [FAuNO: Semi-Asynchronous Federated Reinforcement Learning Framework for Task Offloading in Edge Systems](https://mlsanity.com/paper/2506.02668/) (2025)
- [A Federated Reinforcement Learning Method with Quantization for Cooperative Edge Caching in Fog Radio Access Networks](https://mlsanity.com/paper/2206.11556/) (2022)
- [Privacy-Aware Spectrum Pricing and Power Control Optimization for LEO Satellite Internet-of-Things](https://mlsanity.com/paper/2407.00814/) (2024)
- [Single-Loop Federated Actor-Critic across Heterogeneous Environments](https://mlsanity.com/paper/2412.14555/) (2024)
- [FedQHD: Closed-Form Function-Space Federated Reinforcement Learning](https://mlsanity.com/paper/2605.29002/) (2026)

## Abstract (arXiv)

Ultra-dense edge computing (UDEC) has great potential, especially in the 5G era, but it still faces challenges in its current solutions, such as the lack of: i) efficient utilization of multiple 5G resources (e.g., computation, communication, storage and service resources); ii) low overhead offloading decision making and resource allocation strategies; and iii) privacy and security protection schemes. Thus, we first propose an intelligent ultra-dense edge computing (I-UDEC) framework, which integrates blockchain and Artificial Intelligence (AI) into 5G ultra-dense edge computing networks. First, we show the architecture of the framework. Then, in order to achieve real-time and low overhead computation offloading decisions and resource allocation strategies, we design a novel two-timescale deep reinforcement learning (textit2Ts-DRL) approach, consisting of a fast-timescale and a slow-timescale learning process, respectively. The primary objective is to minimize the total offloading delay and network resource usage by jointly optimizing computation offloading, resource allocation and service caching placement. We also leverage federated learning (FL) to train the textit2Ts-DRL model in a distributed manner, aiming to protect the edge devices' data privacy. Simulation results corroborate the effectiveness of both the textit2Ts-DRL and FL in the I-UDEC framework and prove that our proposed algorithm can reduce task execution time up to 31.87%.