# Muti-Agent Proximal Policy Optimization For Data Freshness in UAV-assisted Networks

arXiv: https://arxiv.org/abs/2303.08680 · HTML: https://mlsanity.com/paper/2303.08680/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networks](https://mlsanity.com/paper/1810.10408/) (2018)
- [Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey](https://mlsanity.com/paper/2110.13484/) (2021)
- [Distributed Multi-agent Meta Learning for Trajectory Design in Wireless Drone Networks](https://mlsanity.com/paper/2012.03158/) (2020)
- [Unmanned Aerial Vehicle with Underlaid Device-to-Device Communications: Performance and Tradeoffs](https://mlsanity.com/paper/1509.01187/) (2015)
- [Age-of-Updates Optimization for UAV-assisted Networks](https://mlsanity.com/paper/2209.05370/) (2022)
- [Aeronautical Data Aggregation and Field Estimation in IoT Networks: Hovering & Traveling Time Dilemma of UAVs](https://mlsanity.com/paper/1810.08035/) (2018)

## Built on by (most cited first, 1 of 1)

- [Ensemble DNN for Age-of-Information Minimization in UAV-assisted Networks](https://mlsanity.com/paper/2309.02913/) (2023)

## Abstract (arXiv)

Unmanned aerial vehicles (UAVs) are seen as a promising technology to perform a wide range of tasks in wireless communication networks. In this work, we consider the deployment of a group of UAVs to collect the data generated by IoT devices. Specifically, we focus on the case where the collected data is time-sensitive, and it is critical to maintain its timeliness. Our objective is to optimally design the UAVs' trajectories and the subsets of visited IoT devices such as the global Age-of-Updates (AoU) is minimized. To this end, we formulate the studied problem as a mixed-integer nonlinear programming (MINLP) under time and quality of service constraints. To efficiently solve the resulting optimization problem, we investigate the cooperative Multi-Agent Reinforcement Learning (MARL) framework and propose an RL approach based on the popular on-policy Reinforcement Learning (RL) algorithm: Policy Proximal Optimization (PPO). Our approach leverages the centralized training decentralized execution (CTDE) framework where the UAVs learn their optimal policies while training a centralized value function. Our simulation results show that the proposed MAPPO approach reduces the global AoU by at least a factor of 1/2 compared to conventional off-policy reinforcement learning approaches.