# Leveraging Digital Cousins for Ensemble Q-Learning in Large-Scale Wireless Networks

arXiv: https://arxiv.org/abs/2402.08022 · HTML: https://mlsanity.com/paper/2402.08022/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 17 of 17)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient](https://mlsanity.com/paper/2210.06718/) (2022)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Edge Learning for B5G Networks with Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing](https://mlsanity.com/paper/2206.00422/) (2022)
- [Digital Twin of Wireless Systems: Overview, Taxonomy, Challenges, and Opportunities](https://mlsanity.com/paper/2202.02559/) (2022)
- [6G Wireless Communication Systems: Applications, Requirements, Technologies, Challenges, and Research Directions](https://mlsanity.com/paper/1909.11315/) (2019)
- [Ensemble Bootstrapping for Q-Learning](https://mlsanity.com/paper/2103.00445/) (2021)
- [Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks](https://mlsanity.com/paper/2011.09902/) (2020)
- [Deep Learning for Hybrid 5G Services in Mobile Edge Computing Systems: Learn from a Digital Twin](https://mlsanity.com/paper/1907.01523/) (2019)
- [AI-enabled Future Wireless Networks: Challenges, Opportunities and Open Issues](https://mlsanity.com/paper/2103.04536/) (2021)
- [Implications of Decentralized Q-learning Resource Allocation in Wireless Networks](https://mlsanity.com/paper/1705.10508/) (2017)
- [DQLEL: Deep Q-Learning for Energy-Optimized LoS/NLoS UWB Node Selection](https://mlsanity.com/paper/2108.13157/) (2021)
- [Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization](https://mlsanity.com/paper/2402.05476/) (2024)

## Built on by (most cited first, 4 of 4)

- [Coverage Analysis of Multi-Environment Q-Learning Algorithms for Wireless Network Optimization](https://mlsanity.com/paper/2408.16882/) (2024)
- [A Multi-Agent Multi-Environment Mixed Q-Learning for Partially Decentralized Wireless Network Optimization](https://mlsanity.com/paper/2409.16450/) (2024)
- [Provable Performance Bounds for Digital Twin-driven Deep Reinforcement Learning in Wireless Networks: A Novel Digital-Twin Bisimulation Metric](https://mlsanity.com/paper/2502.17983/) (2025)
- [Partially Decentralized Multi-Agent Q-Learning via Digital Cousins for Wireless Networks](https://mlsanity.com/paper/2503.05970/) (2025)

## Abstract (arXiv)

Optimizing large-scale wireless networks, including optimal resource management, power allocation, and throughput maximization, is inherently challenging due to their non-observable system dynamics and heterogeneous and complex nature. Herein, a novel ensemble Q-learning algorithm that addresses the performance and complexity challenges of the traditional Q-learning algorithm for optimizing wireless networks is presented. Ensemble learning with synthetic Markov Decision Processes is tailored to wireless networks via new models for approximating large state-space observable wireless networks. In particular, digital cousins are proposed as an extension of the traditional digital twin concept wherein multiple Q-learning algorithms on multiple synthetic Markovian environments are run in parallel and their outputs are fused into a single Q-function. Convergence analyses of key statistics and Q-functions and derivations of upper bounds on the estimation bias and variance are provided. Numerical results across a variety of real-world wireless networks show that the proposed algorithm can achieve up to 50% less average policy error with up to 40% less runtime complexity than the state-of-the-art reinforcement learning algorithms. It is also shown that theoretical results properly predict trends in the experimental results.