A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networks
On the map: Deep reinforcement learning methods → RL for energy, networks and operations
Abstract (arXiv)
As reconfigurable intelligent surfaces (RIS) emerge as a pivotal technology in the upcoming sixth-generation (6G) networks, their deployment within practical multiple operator (OP) networks presents significant challenges, including the coordination of RIS configurations among OPs, interference management, and privacy maintenance. A promising strategy is to treat RIS as a public resource managed by an RIS provider (RP), which can enhance resource allocation efficiency by allowing dynamic access for multiple OPs. However, the intricate nature of coordinating management and optimizing RIS configurations significantly complicates the implementation process. In this paper, we propose a hierarchical deep reinforcement learning (HDRL) approach that decomposes the complicated RIS resource optimization problem into several subtasks. Specifically, a top-level RP-agent is responsible for RIS allocation, while low-level OP-agents control their assigned RISs and handle beamforming, RIS phase-shifts, and user association. By utilizing the semi-Markov decision process (SMDP) theory, we establish a sophisticated interaction mechanism between the RP and OPs, and introduce an advanced hierarchical proximal policy optimization (HPPO) algorithm. Furthermore, we propose an improved sequential-HPPO (S-HPPO) algorithm to address the curse of dimensionality encountered with a single RP-agent. Experimental results validate the stability of the HPPO algorithm across various environmental parameters, demonstrating its superiority over other benchmarks for joint resource optimization. Finally, we conduct a detailed comparative analysis between the proposed S-HPPO and HPPO algorithms, showcasing that the S-HPPO algorithm achieves faster convergence and improved performance in large-scale RIS allocation scenarios.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 42 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
23 of 42 references.
References · 23
- 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
- 2018Intelligent Reflecting Surface Enhanced Wireless Network via Joint Active and Passive Beamformingcited by 46
- 2021Generalized Proximal Policy Optimization with Sample Reusecited by 40
- 2020Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learningcited by 39
- 2020Deep Reinforcement Learning Based Intelligent Reflecting Surface for Secure Wireless Communicationscited by 22
- 2019Machine Learning for Resource Management in Cellular and IoT Networks: Potentials, Current Solutions, and Open Challengescited by 19
- 2023A Survey on Model-based, Heuristic, and Machine Learning Optimization Approaches in RIS-aided Wireless Networkscited by 15
- 2023Hybrid Hierarchical DRL Enabled Resource Allocation for Secure Transmission in Multi-IRS-Assisted Sensing-Enhanced Spectrum Sharing Networkscited by 6
- 2020RIS Enhanced Massive Non-orthogonal Multiple Access Networks: Deployment and Passive Beamforming Designcited by 5
- 2021Self-Organizing mmWave MIMO Cell-Free Networks With Hybrid Beamforming: A Hierarchical DRL-Based Designcited by 3
- 2023Cooperative Hierarchical Deep Reinforcement Learning based Joint Sleep and Power Control in RIS-aided Energy-Efficient RANcited by 3
- 2020Practical Modeling and Beamforming for Intelligent Reflecting Surface Aided Wideband Systemscited by 2
11 more
- 2020Decentralized Beamforming Design for Intelligent Reflecting Surface-enhanced Cell-free Networkscited by 2
- 2020Intelligent reflecting surface enhanced wideband MIMO-OFDM communications: From practical model to reflection optimizationcited by 2
- 2023On the Impact of an IRS on the Out-of-Band Performance in Sub-6 GHz & mmWave Frequenciescited by 2
- 2020Dynamic Network Service Selection in IRS-Assisted Wireless Networks: A Game Theory Approachcited by 1
- 2021Joint Transmit Waveform and Passive Beamforming Design for RIS-Aided DFRC Systemscited by 1
- 2022IRS-assisted Multi-cell Multi-band Systems: Practical Reflection Model and Joint Beamforming Designcited by 1
- 2022Non-Cooperative Resource Management for Intelligent Reflecting Surface Aided Networkscited by 1
- 2022Joint BS-RIS-User Association and Beamforming Design for RIS-assisted Cellular Networkscited by 1
- 2023Combating Inter-Operator Pilot Contamination in Reconfigurable Intelligent Surfaces Assisted Multi-Operator Networkscited by 1
- 2024Distributed IRSs Always Benefit Every Mobile Operatorcited by 1
- 2024Impact of Inter-Operator Interference via Reconfigurable Intelligent Surfacescited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Auction-Based RIS Allocation With DRL: Controlling the Cost-Performance Trade-Off
- 2024An Overview of Machine Learning-Enabled Optimization for Reconfigurable Intelligent Surfaces-Aided 6G Networks: From Reinforcement Learning to Large Language Models
- 2026Parametrized Sharing for Multi-Agent Hybrid DRL for Multiple Multi-Functional RISs-Aided Downlink NOMA Networks
- 2025Beamforming and Resource Allocation for Delay Minimization in RIS-Assisted OFDM Systems
- 2026Multi-Agent DRL for QoS and Energy Optimization in RIS-Enabled Open-RAN Industrial 6G TN/NTN Networks
- 2024Multi-Agent Deep Reinforcement Learning for Energy Efficient Multi-Hop STAR-RIS-Assisted Transmissions
- 2025Deep Reinforcement Learning for Dynamic Resource Allocation in Wireless Networks
- 2026Digital Twin-Assisted Adaptive Multi-Agent DRL for Intelligent Spectrum and Resource Management in Open-RAN UAV-Enabled 6G Networks