# Towards Dynamic Resource Allocation and Client Scheduling in Hierarchical Federated Learning: A Two-Phase Deep Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2406.14910 · HTML: https://mlsanity.com/paper/2406.14910/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 58 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 22 of 22)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Learning in Mobile Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/1909.11875/) (2019)
- [Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge](https://mlsanity.com/paper/1804.08333/) (2018)
- [A Joint Learning and Communications Framework for Federated Learning over Wireless Networks](https://mlsanity.com/paper/1909.07972/) (2019)
- [Client-Edge-Cloud Hierarchical Federated Learning](https://mlsanity.com/paper/1905.06641/) (2019)
- [Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)](https://mlsanity.com/paper/1812.11494/) (2018)
- [Scheduling Policies for Federated Learning in Wireless Networks](https://mlsanity.com/paper/1908.06287/) (2019)
- [Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation](https://mlsanity.com/paper/1910.13067/) (2019)
- [HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learning](https://mlsanity.com/paper/2002.11343/) (2020)
- [Energy-Efficient Resource Management for Federated Edge Learning with CPU-GPU Heterogeneous Computing](https://mlsanity.com/paper/2007.07122/) (2020)
- [Machine Learning Model Sizes and the Parameter Gap](https://mlsanity.com/paper/2207.02852/) (2022)
- [Energy-Efficient Federated Edge Learning with Joint Communication and Computation Design](https://mlsanity.com/paper/2003.00199/) (2020)
- [Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications](https://mlsanity.com/paper/2012.01489/) (2020)
- [Context-Aware Online Client Selection for Hierarchical Federated Learning](https://mlsanity.com/paper/2112.00925/) (2021)
- [Federated Learning over Energy Harvesting Wireless Networks](https://mlsanity.com/paper/2106.08809/) (2021)
- [Edge Federated Learning Via Unit-Modulus Over-The-Air Computation](https://mlsanity.com/paper/2101.12051/) (2021)
- [Joint Client Scheduling and Resource Allocation under Channel Uncertainty in Federated Learning](https://mlsanity.com/paper/2106.06796/) (2021)
- [Communication-Efficient Hierarchical Federated Learning for IoT Heterogeneous Systems with Imbalanced Data](https://mlsanity.com/paper/2107.06548/) (2021)
- [Time-sensitive Learning for Heterogeneous Federated Edge Intelligence](https://mlsanity.com/paper/2301.10977/) (2023)
- [Device Scheduling and Assignment in Hierarchical Federated Learning for Internet of Things](https://mlsanity.com/paper/2402.02506/) (2024)
- [Meta-material Sensor Based Internet of Things: Design, Optimization, and Implementation](https://mlsanity.com/paper/2206.12894/) (2022)

## Built on by (most cited first, 6 of 6)

- [AI-driven Wireless Positioning: Fundamentals, Standards, State-of-the-art, and Challenges](https://mlsanity.com/paper/2501.14970/) (2025)
- [Adaptive UAV-Assisted Hierarchical Federated Learning: Optimizing Energy, Latency, and Resilience for Dynamic Smart IoT](https://mlsanity.com/paper/2503.06145/) (2025)
- [Age-Aware Edge-Blind Federated Learning via Over-the-Air Aggregation](https://mlsanity.com/paper/2602.02469/) (2026)
- [Towards Seamless Hierarchical Federated Learning under Intermittent Client Participation: A Stagewise Decision-Making Methodology](https://mlsanity.com/paper/2502.09303/) (2025)
- [Joint Optimization of Training and Inference in Federated Edge Learning via Constrained Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/2605.25916/) (2026)
- [Geometric Fairness-Aware Routing for Federated Edge Networks](https://mlsanity.com/paper/2606.26125/) (2026)

## Abstract (arXiv)

Federated learning (FL) is a viable technique to train a shared machine learning model without sharing data. Hierarchical FL (HFL) system has yet to be studied regrading its multiple levels of energy, computation, communication, and client scheduling, especially when it comes to clients relying on energy harvesting to power their operations. This paper presents a new two-phase deep deterministic policy gradient (DDPG) framework, referred to as ``TP-DDPG'', to balance online the learning delay and model accuracy of an FL process in an energy harvesting-powered HFL system. The key idea is that we divide optimization decisions into two groups, and employ DDPG to learn one group in the first phase, while interpreting the other group as part of the environment to provide rewards for training the DDPG in the second phase. Specifically, the DDPG learns the selection of participating clients, and their CPU configurations and the transmission powers. A new straggler-aware client association and bandwidth allocation (SCABA) algorithm efficiently optimizes the other decisions and evaluates the reward for the DDPG. Experiments demonstrate that with substantially reduced number of learnable parameters, the TP-DDPG can quickly converge to effective polices that can shorten the training time of HFL by 39.4% compared to its benchmarks, when the required test accuracy of HFL is 0.9.