# Heterogeneous Multi-Agent Reinforcement Learning for Radio Resource Management under Coupled Finite-Horizon Constraints

arXiv: https://arxiv.org/abs/2608.01745 · HTML: https://mlsanity.com/paper/2608.01745/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 8 of 8)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks](https://mlsanity.com/paper/1808.00490/) (2018)
- [Multi-Agent Constrained Policy Optimisation](https://mlsanity.com/paper/2110.02793/) (2021)
- [Multi-Agent Reinforcement Learning for Adaptive User Association in Dynamic mmWave Networks](https://mlsanity.com/paper/2006.09066/) (2020)
- [A Survey of Energy-Efficient Techniques for 5G Networks and Challenges Ahead](https://mlsanity.com/paper/1604.00786/) (2016)
- [Multi-Agent Reinforcement Learning for Multi-Cell Spectrum and Power Allocation](https://mlsanity.com/paper/2312.05746/) (2023)
- [An Offline Multi-Agent Reinforcement Learning Framework for Radio Resource Management](https://mlsanity.com/paper/2501.12991/) (2025)

## Abstract (arXiv)

Maximizing throughput under proportional fairness in dense wireless networks requires jointly managing user association, scheduling, base station (BS) activation, and handover control under hard finite-horizon energy and handover budgets, which induces a fundamental tension between BS-side energy management and user-side handover regulation. While multi-agent reinforcement learning (MARL) is a natural framework for such distributed sequential control, its application here faces two difficulties: finite-horizon budget constraints cannot be evaluated at each time slot, and the nonlinear proportional fairness utility admits no principled per-slot decomposition. We propose HeLyMARL, a Lyapunov-embedded heterogeneous MARL framework that resolves both via drift-plus-penalty decomposition with virtual queues. The energy and handover constraint pressures are internalized directly into a unified per-slot reward, converting the constrained finite-horizon problem into an unconstrained MARL problem. Comparison against two Lagrangian-based alternatives reveals a timescale separation: Lagrangian relaxation regulates constraints only across training episodes, whereas the virtual queues of HeLyMARL bound cumulative budget consumption at every partial horizon within an episode, a pacing guarantee beyond the reach of greedy Lyapunov-based control. Simulations show that HeLyMARL is the only method that sustains the throughput-fairness balance together with uninterrupted service throughout the horizon, outperforming conventional MARL, Lyapunov-based, and constrained MARL benchmarks without premature budget exhaustion.