# Dynamic Trajectory and Power Control in Ultra-Dense UAV Networks: A Mean-Field Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2411.14052 · HTML: https://mlsanity.com/paper/2411.14052/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 15 of 15)

- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networks](https://mlsanity.com/paper/1810.10408/) (2018)
- [UAV Communications for 5G and Beyond: Recent Advances and Future Trends](https://mlsanity.com/paper/1901.06637/) (2019)
- [Multi-Agent Deep Reinforcement Learning Based Trajectory Planning for Multi-UAV Assisted Mobile Edge Computing](https://mlsanity.com/paper/2009.11277/) (2020)
- [Path Planning for UAV-Mounted Mobile Edge Computing with Deep Reinforcement Learning](https://mlsanity.com/paper/2001.10268/) (2020)
- [Decision Transformers for Wireless Communications: A New Paradigm of Resource Management](https://mlsanity.com/paper/2404.05199/) (2024)
- [Cellular-Connected UAVs over 5G: Deep Reinforcement Learning for Interference Management](https://mlsanity.com/paper/1801.05500/) (2018)
- [A Survey of Channel Modeling for UAV Communications](https://mlsanity.com/paper/1801.07359/) (2018)
- [Optimal 3D-Trajectory Design and Resource Allocation for Solar-Powered UAV Communication Systems](https://mlsanity.com/paper/1808.00101/) (2018)
- [UAV Communications Based on Non-Orthogonal Multiple Access](https://mlsanity.com/paper/1809.05767/) (2018)
- [Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Control](https://mlsanity.com/paper/2211.02296/) (2022)
- [Massive UAV-to-Ground Communication and its Stable Movement Control: A Mean-Field Approach](https://mlsanity.com/paper/1803.03285/) (2018)
- [Probabilistic Caching for Small-Cell Networks with Terrestrial and Aerial Users](https://mlsanity.com/paper/2003.13201/) (2020)

## Built on by (most cited first, 2 of 2)

- [Embedded Mean Field Reinforcement Learning for Perimeter-defense Game](https://mlsanity.com/paper/2505.14209/) (2025)
- [Green Emergency Communications in RIS- and MA-Assisted Multi-UAV SAGINs: A Partially Observable Reinforcement Learning Approach](https://mlsanity.com/paper/2511.12892/) (2025)

## Abstract (arXiv)

In ultra-dense unmanned aerial vehicle (UAV) networks, it is challenging to coordinate the resource allocation and interference management among large-scale UAVs, for providing flexible and efficient service coverage to the ground users (GUs). In this paper, we propose a learning-based resource allocation scheme in an ultra-dense UAV communication network, where the GUs' service demands are time-varying with unknown distributions. We formulate the non-cooperative game among multiple co-channel UAVs as a stochastic game, where each UAV jointly optimizes its trajectory, user association, and downlink power control to maximize the expectation of its locally cumulative energy efficiency under the interference and energy constraints. To cope with the scalability issue in a large-scale network, we further formulate the problem as a mean-field game (MFG), which simplifies the interactions among the UAVs into a two-player game between a representative UAV and a mean-field. We prove the existence and uniqueness of the equilibrium for the MFG, and propose a model-free mean-field reinforcement learning algorithm named maximum entropy mean-field deep Q network (ME-MFDQN) to solve the mean-field equilibrium in both fully and partially observable scenarios. The simulation results reveal that the proposed algorithm improves the energy efficiency compared with the benchmark algorithms. Moreover, the performance can be further enhanced if the GUs' service demands exhibit higher temporal correlation or if the UAVs have wider observation capabilities over their nearby GUs.