# Asynchronous Fractional Multi-Agent Deep Reinforcement Learning for Age-Minimal Mobile Edge Computing

arXiv: https://arxiv.org/abs/2409.16832 · HTML: https://mlsanity.com/paper/2409.16832/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 84 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Age of Information: An Introduction and Survey](https://mlsanity.com/paper/2007.08564/) (2020)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Deep Reinforcement Learning for Task Offloading in Mobile Edge Computing Systems](https://mlsanity.com/paper/2005.02459/) (2020)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [Deep Q-Learning for Nash Equilibria: Nash-DQN](https://mlsanity.com/paper/1904.10554/) (2019)
- [Dynamic Scheduling for Stochastic Edge-Cloud Computing Environments using A3C learning and Residual Recurrent Neural Networks](https://mlsanity.com/paper/2009.02186/) (2020)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [Survey on Multi-Access Edge Computing for Internet of Things Realization](https://mlsanity.com/paper/1805.06695/) (2018)
- [Sampling for Data Freshness Optimization: Non-linear Age Functions](https://mlsanity.com/paper/1812.07241/) (2018)
- [Age of Information-Aware Radio Resource Management in Vehicular Networks: A Proactive Deep Reinforcement Learning Perspective](https://mlsanity.com/paper/1908.02047/) (2019)
- [Distributed Reinforcement Learning for Privacy-Preserving Dynamic Edge Caching](https://mlsanity.com/paper/2110.10349/) (2021)
- [A Reinforcement Learning Approach to Age of Information in Multi-User Networks with HARQ](https://mlsanity.com/paper/2102.09774/) (2021)
- [Age of Information Aware VNF Scheduling in Industrial IoT Using Deep Reinforcement Learning](https://mlsanity.com/paper/2105.04207/) (2021)
- [Reducing Bus Bunching with Asynchronous Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2105.00376/) (2021)
- [UAV-to-Device Underlay Communications: Age of Information Minimization by Multi-agent Deep Reinforcement Learning](https://mlsanity.com/paper/2003.05830/) (2020)
- [Decentralized Task Offloading in Edge Computing: A Multi-User Multi-Armed Bandit Approach](https://mlsanity.com/paper/2112.11818/) (2021)
- [Age-Delay Tradeoffs in Queueing Systems](https://mlsanity.com/paper/1911.05601/) (2019)
- [Information Freshness-Aware Task Offloading in Air-Ground Integrated Edge Computing Systems](https://mlsanity.com/paper/2007.10129/) (2020)
- [Timely Cloud Computing: Preemption and Waiting](https://mlsanity.com/paper/1907.05408/) (2019)
- [Age of Information for Multicast Transmission with Fixed and Random Deadlines in IoT Systems](https://mlsanity.com/paper/2003.08304/) (2020)
- [Fractional Deep Reinforcement Learning for Age-Minimal Mobile Edge Computing](https://mlsanity.com/paper/2312.10418/) (2023)

## Abstract (arXiv)

In the realm of emerging real-time networked applications such as cyber-physical systems (CPS), the Age of Information (AoI) has emerged as a pivotal metric for evaluating timeliness. To meet the high computational demands, such as those in smart manufacturing within CPS, mobile edge computing (MEC) presents a promising solution for optimizing computing and reducing AoI. In this work, we study the timeliness of compute-intensive updates and explore jointly optimizing the task updating (when to generate a task) and offloading (where to process a task) policies to minimize AoI. Specifically, we consider edge load dynamics and formulate a task scheduling problem to minimize the expected time-average AoI. Solving this problem is challenging due to the fractional objective introduced by AoI and the asynchronous decision-making of the semi-Markov game (SMG). To this end, we propose a fractional reinforcement learning (RL) framework. We begin by introducing a fractional single-agent RL framework and establish its linear convergence rate. Building on this, we develop a fractional multi-agent RL framework, extend Dinkelbach's method, and demonstrate its equivalence to the inexact Newton's method. Furthermore, we provide the conditions under which the framework achieves linear convergence to the Nash equilibrium (NE). To tackle the challenge of asynchronous decision-making in the SMG, we further design an asynchronous model-free fractional multi-agent RL algorithm, where each mobile device can determine the task updating and offloading decisions without knowing the real-time system dynamics and decisions of other devices. Experimental results show that when compared with the best existing baseline algorithm, our proposed algorithm reduces the average AoI by up to 50.6%.