# Learning to Reflect: Hierarchical Multi-Agent Reinforcement Learning for CSI-Free mmWave Beam-Focusing

arXiv: https://arxiv.org/abs/2603.07370 · HTML: https://mlsanity.com/paper/2603.07370/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 7 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Sionna: An Open-Source Library for Next-Generation Physical Layer Research](https://mlsanity.com/paper/2203.11854/) (2022)
- [Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learning](https://mlsanity.com/paper/2002.10072/) (2020)
- [Enabling Large Intelligent Surfaces with Compressive Sensing and Deep Learning](https://mlsanity.com/paper/1904.10136/) (2019)
- [HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism](https://mlsanity.com/paper/2110.07246/) (2021)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [Pervasive Machine Learning for Smart Radio Environments Enabled by Reconfigurable Intelligent Surfaces](https://mlsanity.com/paper/2205.03793/) (2022)
- [Deep Reinforcement Learning for Intelligent Reflecting Surfaces: Towards Standalone Operation](https://mlsanity.com/paper/2002.11101/) (2020)
- [Multi-Player Zero-Sum Markov Games with Networked Separable Interactions](https://mlsanity.com/paper/2307.09470/) (2023)
- [Two-Timescale Channel Estimation for Reconfigurable Intelligent Surface Aided Wireless Communications](https://mlsanity.com/paper/1912.07990/) (2019)
- [Active RIS vs. Passive RIS: Which Will Prevail in 6G?](https://mlsanity.com/paper/2103.15154/) (2021)
- [An Overview of Signal Processing Techniques for RIS/IRS-aided Wireless Systems](https://mlsanity.com/paper/2112.05989/) (2021)
- [Coverage Enhancement for NLOS mmWave Links Using Passive Reflectors](https://mlsanity.com/paper/1905.04794/) (2019)
- [Channel Estimation for Reconfigurable Intelligent Surface Aided Multi-User mmWave MIMO Systems](https://mlsanity.com/paper/1912.03619/) (2019)
- [Reconfigurable Intelligent Surfaces: A Signal Processing Perspective With Wireless Applications](https://mlsanity.com/paper/2102.00742/) (2021)
- [A Deep Reinforcement Learning Approach for Autonomous Reconfigurable Intelligent Surfaces](https://mlsanity.com/paper/2403.09270/) (2024)
- [Insight Gained from Migrating a Machine Learning Model to Intelligence Processing Units](https://mlsanity.com/paper/2404.10730/) (2024)
- [Guiding Wireless Signals with Arrays of Metallic Linear Fresnel Reflectors: A Low-cost, Frequency-versatile, and Practical Approach](https://mlsanity.com/paper/2407.19179/) (2024)
- [Signal Whisperers: Enhancing Wireless Reception Using DRL-Guided Reflector Arrays](https://mlsanity.com/paper/2501.15044/) (2025)
- [Practical Channel Estimation and Phase Shift Design for Intelligent Reflecting Surface Empowered MIMO Systems](https://mlsanity.com/paper/2104.14161/) (2021)
- [Codebook-Based Solutions for Reconfigurable Intelligent Surfaces and Their Open Challenges](https://mlsanity.com/paper/2211.05976/) (2022)
- [Wireless Communication Using Metal Reflectors: Reflection Modelling and Experimental Verification](https://mlsanity.com/paper/2211.08626/) (2022)
- [Channel Training and Estimation for Reconfigurable Intelligent Surfaces: Exposition of Principles, Approaches, and Open Problems](https://mlsanity.com/paper/2301.05026/) (2023)

## Abstract (arXiv)

Reconfigurable Intelligent Surfaces promise to transform wireless environments, yet practical deployment is hindered by the prohibitive overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization. This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework for the control of mechanically reconfigurable reflective surfaces in millimeter-wave (mmWave) systems. We introduce a "CSI-free" paradigm that substitutes pilot-based channel estimation with readily available user localization data. To manage the massive combinatorial action space, the proposed architecture utilizes Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) paradigm. The proposed architecture decomposes the control problem into two abstraction levels: a high-level controller for user-to-reflector allocation and decentralized low-level controllers for low-level focal point optimization. Comprehensive ray-tracing evaluations demonstrate that the framework achieves 2.81-7.94 dB RSSI improvements over centralized baselines, with the performance advantage widening as system complexity increases. Scalability analysis reveals that the system maintains sustained efficiency, exhibiting minimal per-user performance degradation and stable total power utilization even when user density doubles. Furthermore, robustness validation confirms the framework's viability across varying reflector aperture sizes (45-99 tiles) and demonstrates graceful performance degradation under localization errors up to 0.5 m. By eliminating CSI overhead while maintaining high-fidelity beam-focusing, this work establishes HMARL as a practical solution for intelligent mmWave environments.