# A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networks arXiv: https://arxiv.org/abs/2410.12320 ยท HTML: https://mlsanity.com/paper/2410.12320/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. - Builds on 23 of its 42 references that are in the map. - First posted on arXiv 16 Oct 2024. ## Placed in - Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/) ## Builds on (most cited first, 23 of 23) - [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017) - [Intelligent Reflecting Surface Enhanced Wireless Network via Joint Active and Passive Beamforming](https://mlsanity.com/paper/1810.03961/) (2018) - [Generalized Proximal Policy Optimization with Sample Reuse](https://mlsanity.com/paper/2111.00072/) (2021) - [Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learning](https://mlsanity.com/paper/2002.10072/) (2020) - [Deep Reinforcement Learning Based Intelligent Reflecting Surface for Secure Wireless Communications](https://mlsanity.com/paper/2002.12271/) (2020) - [Machine Learning for Resource Management in Cellular and IoT Networks: Potentials, Current Solutions, and Open Challenges](https://mlsanity.com/paper/1907.08965/) (2019) - [A Survey on Model-based, Heuristic, and Machine Learning Optimization Approaches in RIS-aided Wireless Networks](https://mlsanity.com/paper/2303.14320/) (2023) - [Hybrid Hierarchical DRL Enabled Resource Allocation for Secure Transmission in Multi-IRS-Assisted Sensing-Enhanced Spectrum Sharing Networks](https://mlsanity.com/paper/2312.01071/) (2023) - [RIS Enhanced Massive Non-orthogonal Multiple Access Networks: Deployment and Passive Beamforming Design](https://mlsanity.com/paper/2001.10363/) (2020) - [Self-Organizing mmWave MIMO Cell-Free Networks With Hybrid Beamforming: A Hierarchical DRL-Based Design](https://mlsanity.com/paper/2103.11823/) (2021) - [Cooperative Hierarchical Deep Reinforcement Learning based Joint Sleep and Power Control in RIS-aided Energy-Efficient RAN](https://mlsanity.com/paper/2304.13226/) (2023) - [Practical Modeling and Beamforming for Intelligent Reflecting Surface Aided Wideband Systems](https://mlsanity.com/paper/2006.01465/) (2020) - [Decentralized Beamforming Design for Intelligent Reflecting Surface-enhanced Cell-free Networks](https://mlsanity.com/paper/2006.12238/) (2020) - [Intelligent reflecting surface enhanced wideband MIMO-OFDM communications: From practical model to reflection optimization](https://mlsanity.com/paper/2007.14243/) (2020) - [On the Impact of an IRS on the Out-of-Band Performance in Sub-6 GHz & mmWave Frequencies](https://mlsanity.com/paper/2308.14033/) (2023) - [Dynamic Network Service Selection in IRS-Assisted Wireless Networks: A Game Theory Approach](https://mlsanity.com/paper/2008.01239/) (2020) - [Joint Transmit Waveform and Passive Beamforming Design for RIS-Aided DFRC Systems](https://mlsanity.com/paper/2112.08861/) (2021) - [IRS-assisted Multi-cell Multi-band Systems: Practical Reflection Model and Joint Beamforming Design](https://mlsanity.com/paper/2204.06351/) (2022) - [Non-Cooperative Resource Management for Intelligent Reflecting Surface Aided Networks](https://mlsanity.com/paper/2209.00226/) (2022) - [Joint BS-RIS-User Association and Beamforming Design for RIS-assisted Cellular Networks](https://mlsanity.com/paper/2210.17260/) (2022) - [Combating Inter-Operator Pilot Contamination in Reconfigurable Intelligent Surfaces Assisted Multi-Operator Networks](https://mlsanity.com/paper/2311.01151/) (2023) - [Distributed IRSs Always Benefit Every Mobile Operator](https://mlsanity.com/paper/2401.08434/) (2024) - [Impact of Inter-Operator Interference via Reconfigurable Intelligent Surfaces](https://mlsanity.com/paper/2403.00349/) (2024) ## Built on by (most cited first, 1 of 1) - [Cooperative Deep Reinforcement Learning for Fair RIS Allocation](https://mlsanity.com/paper/2603.25572/) (2026) ## Abstract (arXiv) As reconfigurable intelligent surfaces (RIS) emerge as a pivotal technology in the upcoming sixth-generation (6G) networks, their deployment within practical multiple operator (OP) networks presents significant challenges, including the coordination of RIS configurations among OPs, interference management, and privacy maintenance. A promising strategy is to treat RIS as a public resource managed by an RIS provider (RP), which can enhance resource allocation efficiency by allowing dynamic access for multiple OPs. However, the intricate nature of coordinating management and optimizing RIS configurations significantly complicates the implementation process. In this paper, we propose a hierarchical deep reinforcement learning (HDRL) approach that decomposes the complicated RIS resource optimization problem into several subtasks. Specifically, a top-level RP-agent is responsible for RIS allocation, while low-level OP-agents control their assigned RISs and handle beamforming, RIS phase-shifts, and user association. By utilizing the semi-Markov decision process (SMDP) theory, we establish a sophisticated interaction mechanism between the RP and OPs, and introduce an advanced hierarchical proximal policy optimization (HPPO) algorithm. Furthermore, we propose an improved sequential-HPPO (S-HPPO) algorithm to address the curse of dimensionality encountered with a single RP-agent. Experimental results validate the stability of the HPPO algorithm across various environmental parameters, demonstrating its superiority over other benchmarks for joint resource optimization. Finally, we conduct a detailed comparative analysis between the proposed S-HPPO and HPPO algorithms, showcasing that the S-HPPO algorithm achieves faster convergence and improved performance in large-scale RIS allocation scenarios.