# Collaborative Intelligent Reflecting Surface Networks with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2203.14152 · HTML: https://mlsanity.com/paper/2203.14152/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space](https://mlsanity.com/paper/1810.06394/) (2018)
- [On Safeguarding Privacy and Security in the Framework of Federated Learning](https://mlsanity.com/paper/1909.06512/) (2019)
- [Intelligent Reflecting Surface Enhanced Wireless Network via Joint Active and Passive Beamforming](https://mlsanity.com/paper/1810.03961/) (2018)
- [Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learning](https://mlsanity.com/paper/2002.10072/) (2020)
- [Towards Smart and Reconfigurable Environment: Intelligent Reflecting Surface Aided Wireless Network](https://mlsanity.com/paper/1905.00152/) (2019)
- [Intelligent Reflecting Surface Aided Wireless Communications: A Tutorial](https://mlsanity.com/paper/2007.02759/) (2020)
- [Deep Reinforcement Learning Based Intelligent Reflecting Surface for Secure Wireless Communications](https://mlsanity.com/paper/2002.12271/) (2020)
- [Path Planning for UAV-Mounted Mobile Edge Computing with Deep Reinforcement Learning](https://mlsanity.com/paper/2001.10268/) (2020)
- [Five Disruptive Technology Directions for 5G](https://mlsanity.com/paper/1312.0229/) (2013)
- [Intelligent Reflecting Surface: Practical Phase Shift Model and Beamforming Optimization](https://mlsanity.com/paper/1907.06002/) (2019)
- [Indoor Signal Focusing with Deep Learning Designed Reconfigurable Intelligent Surfaces](https://mlsanity.com/paper/1905.07726/) (2019)
- [Beamforming Optimization for Wireless Network Aided by Intelligent Reflecting Surface with Discrete Phase Shifts](https://mlsanity.com/paper/1906.03165/) (2019)
- [Deep Reinforcement Learning for Intelligent Reflecting Surfaces: Towards Standalone Operation](https://mlsanity.com/paper/2002.11101/) (2020)
- [Multi-hop RIS-Empowered Terahertz Communications: A DRL-based Hybrid Beamforming Design](https://mlsanity.com/paper/2101.09137/) (2021)
- [Intelligent Reflecting Surfaces: Physics, Propagation, and Pathloss Modeling](https://mlsanity.com/paper/1911.03359/) (2019)
- [Intelligent Reflecting Surface: Practical Phase Shift Model and Beamforming Optimization](https://mlsanity.com/paper/2002.10112/) (2020)
- [AI for 5G: Research Directions and Paradigms](https://mlsanity.com/paper/1807.08671/) (2018)
- [Channel Estimation and Passive Beamforming for Intelligent Reflecting Surface: Discrete Phase Shift and Progressive Refinement](https://mlsanity.com/paper/1912.10646/) (2019)
- [A Framework of Robust Transmission Design for IRS-aided MISO Communications with Imperfect Cascaded Channels](https://mlsanity.com/paper/2001.07054/) (2020)
- [UAV-Assisted Intelligent Reflecting Surface Symbiotic Radio System](https://mlsanity.com/paper/2007.14029/) (2020)
- [Joint Active and Passive Beamforming Optimization for Intelligent Reflecting Surface Assisted SWIPT under QoS Constraints](https://mlsanity.com/paper/1910.06220/) (2019)
- [Passive Beamforming and Information Transfer via Large Intelligent Surface](https://mlsanity.com/paper/1905.01491/) (2019)
- [Intelligent Reflecting Surface Aided MIMO Broadcasting for Simultaneous Wireless Information and Power Transfer](https://mlsanity.com/paper/1908.04863/) (2019)
- [Enhancing Cellular Communications for UAVs via Intelligent Reflective Surface](https://mlsanity.com/paper/1911.07631/) (2019)
- [IRS-Assisted Millimeter Wave Communications: Joint Power Allocation and Beamforming Design](https://mlsanity.com/paper/2001.07467/) (2020)

## Built on by (most cited first, 1 of 1)

- [Visual Reasoning and Multi-Agent Approach in Multimodal Large Language Models (MLLMs): Solving TSP and mTSP Combinatorial Challenges](https://mlsanity.com/paper/2407.00092/) (2024)

## Abstract (arXiv)

Intelligent reflecting surface (IRS) is envisioned to be widely applied in future wireless networks. In this paper, we investigate a multi-user communication system assisted by cooperative IRS devices with the capability of energy harvesting. Aiming to maximize the long-term average achievable system rate, an optimization problem is formulated by jointly designing the transmit beamforming at the base station (BS) and discrete phase shift beamforming at the IRSs, with the constraints on transmit power, user data rate requirement and IRS energy buffer size. Considering time-varying channels and stochastic arrivals of energy harvested by the IRSs, we first formulate the problem as a Markov decision process (MDP) and then develop a novel multi-agent Q-mix (MAQ) framework with two layers to decouple the optimization parameters. The higher layer is for optimizing phase shift resolutions, and the lower one is for phase shift beamforming and power allocation. Since the phase shift optimization is an integer programming problem with a large-scale action space, we improve MAQ by incorporating the Wolpertinger method, namely, MAQ-WP algorithm to achieve a sub-optimality with reduced dimensions of action space. In addition, as MAQ-WP is still of high complexity to achieve good performance, we propose a policy gradient-based MAQ algorithm, namely, MAQ-PG, by mapping the discrete phase shift actions into a continuous space at the cost of a slight performance loss. Simulation results demonstrate that the proposed MAQ-WP and MAQ-PG algorithms can converge faster and achieve data rate improvements of 10.7% and 8.8% over the conventional multi-agent DDPG, respectively.