# Partially Decentralized Multi-Agent Q-Learning via Digital Cousins for Wireless Networks

arXiv: https://arxiv.org/abs/2503.05970 · HTML: https://mlsanity.com/paper/2503.05970/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [A Survey of Multi-Agent Deep Reinforcement Learning with Communication](https://mlsanity.com/paper/2203.08975/) (2022)
- [Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial](https://mlsanity.com/paper/2011.03615/) (2020)
- [Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey](https://mlsanity.com/paper/2110.13484/) (2021)
- [Distributed Multi-agent Meta Learning for Trajectory Design in Wireless Drone Networks](https://mlsanity.com/paper/2012.03158/) (2020)
- [Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization](https://mlsanity.com/paper/2006.00587/) (2020)
- [Generative AI for Deep Reinforcement Learning: Framework, Analysis, and Use Cases](https://mlsanity.com/paper/2405.20568/) (2024)
- [Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization](https://mlsanity.com/paper/2402.05476/) (2024)
- [Leveraging Digital Cousins for Ensemble Q-Learning in Large-Scale Wireless Networks](https://mlsanity.com/paper/2402.08022/) (2024)
- [Coverage Analysis of Multi-Environment Q-Learning Algorithms for Wireless Network Optimization](https://mlsanity.com/paper/2408.16882/) (2024)
- [A Multi-Agent Multi-Environment Mixed Q-Learning for Partially Decentralized Wireless Network Optimization](https://mlsanity.com/paper/2409.16450/) (2024)
- [Coverage Analysis for Digital Cousin Selection -- Improving Multi-Environment Q-Learning](https://mlsanity.com/paper/2411.08360/) (2024)

## Abstract (arXiv)

Q-learning is a widely used reinforcement learning (RL) algorithm for optimizing wireless networks, but faces challenges with large state-spaces. Recently proposed multi-environment mixed Q-learning (MEMQ) algorithm addresses these challenges by employing multiple Q-learning algorithms across multiple synthetically generated, distinct but structurally related environments, so-called digital cousins. In this paper, we propose a novel multi-agent MEMQ (M-MEMQ) for cooperative decentralized wireless networks with multiple networked transmitters (TXs) and base stations (BSs). TXs do not have access to global information (joint state and actions). The new concept of coordinated and uncoordinated states is introduced. In uncoordinated states, TXs act independently to minimize their individual costs and update local Q-functions. In coordinated states, TXs use a Bayesian approach to estimate the joint state and update the joint Q-functions. The cost of information-sharing scales linearly with the number of TXs and is independent of the joint state-action space size. Several theoretical guarantees, including deterministic and probabilistic convergence, bounds on estimation error variance, and the probability of misdetecting the joint states, are given. Numerical simulations show that M-MEMQ outperforms several decentralized and centralized training with decentralized execution (CTDE) multi-agent RL algorithms by achieving 60% lower average policy error (APE), 40% faster convergence, 45% reduced runtime complexity, and 40% less sample complexity. Furthermore, M-MEMQ achieves comparable APE with significantly lower complexity than centralized methods. Simulations validate the theoretical analyses.