# Efficient Beam Selection for ISAC in Cell-Free Massive MIMO via Digital Twin-Assisted Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.18560 · HTML: https://mlsanity.com/paper/2506.18560/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 23 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)
- [Integrated Sensing and Communications: Towards Dual-functional Wireless Networks for 6G and Beyond](https://mlsanity.com/paper/2108.07165/) (2021)
- [A Tutorial on Beam Management for 3GPP NR at mmWave Frequencies](https://mlsanity.com/paper/1804.01908/) (2018)
- [Real-Time Digital Twins: Vision and Research Directions for 6G and Beyond](https://mlsanity.com/paper/2301.11283/) (2023)
- [An Overview of Signal Processing Techniques for Joint Communication and Radar Sensing](https://mlsanity.com/paper/2102.12780/) (2021)
- [Deep Learning for Hybrid 5G Services in Mobile Edge Computing Systems: Learn from a Digital Twin](https://mlsanity.com/paper/1907.01523/) (2019)
- [Integrated Sensing and Communication with Massive MIMO: A Unified Tensor Approach for Channel and Target Parameter Estimation](https://mlsanity.com/paper/2401.01738/) (2024)
- [Learning-based Predictive Beamforming for Integrated Sensing and Communication in Vehicular Networks](https://mlsanity.com/paper/2108.11540/) (2021)
- [Digital Twin-Enhanced Deep Reinforcement Learning for Resource Management in Networks Slicing](https://mlsanity.com/paper/2311.16876/) (2023)
- [Joint Deep Reinforcement Learning and Unfolding: Beam Selection and Precoding for mmWave Multiuser MIMO with Lens Arrays](https://mlsanity.com/paper/2101.01336/) (2021)
- [Integrated Sensing and Communications for Low-Altitude Economy: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2412.04074/) (2024)
- [Cell-Free ISAC MIMO Systems: Joint Sensing and Communication Beamforming](https://mlsanity.com/paper/2301.11328/) (2023)
- [Terahertz Integrated Sensing and Communication-Empowered UAVs in 6G: A Transceiver Design Perspective](https://mlsanity.com/paper/2502.04877/) (2025)
- [Multi-Static Target Detection and Power Allocation for Integrated Sensing and Communication in Cell-Free Massive MIMO](https://mlsanity.com/paper/2305.12523/) (2023)
- [Cooperative ISAC-empowered Low-Altitude Economy](https://mlsanity.com/paper/2412.20371/) (2024)
- [Optimal Coordinated Transmit Beamforming for Networked Integrated Sensing and Communications](https://mlsanity.com/paper/2307.05127/) (2023)
- [Communication-Sensing Region for Cell-Free Massive MIMO ISAC Systems](https://mlsanity.com/paper/2307.16259/) (2023)
- [Multi-Static ISAC in Cell-Free Massive MIMO: Precoder Design and Privacy Assessment](https://mlsanity.com/paper/2309.13368/) (2023)
- [Energy-Efficient Hybrid Beamforming with Dynamic On-off Control for Integrated Sensing, Communications, and Powering](https://mlsanity.com/paper/2403.16353/) (2024)
- [Multiple-Target Detection in Cell-Free Massive MIMO-Assisted ISAC](https://mlsanity.com/paper/2404.17263/) (2024)
- [Downlink Beamforming for Cell-Free ISAC: A Fast Complex Oblique Manifold Approach](https://mlsanity.com/paper/2409.06847/) (2024)

## Abstract (arXiv)

Beamforming enhances signal strength and quality by focusing energy in specific directions. This capability is particularly crucial in cell-free integrated sensing and communication (ISAC) systems, where multiple distributed access points (APs) collaborate to provide both communication and sensing services. In this work, we first derive the distribution of joint target detection probabilities across multiple receiving APs under false alarm rate constraints, and then formulate the beam selection procedure as a Markov decision process (MDP). We establish a deep reinforcement learning (DRL) framework, in which reward shaping and sinusoidal embedding are introduced to facilitate agent learning. To eliminate the high costs and associated risks of real-time agent-environment interactions, we further propose a novel digital twin (DT)-assisted offline DRL approach. Different from traditional online DRL, a conditional generative adversarial network (cGAN)-based DT module, operating as a replica of the real world, is meticulously designed to generate virtual state-action transition pairs and enrich data diversity, enabling offline adjustment of the agent's policy. Additionally, we address the out-of-distribution issue by incorporating an extra penalty term into the loss function design. The convergency of agent-DT interaction and the upper bound of the Q-error function are theoretically derived. Numerical results demonstrate the remarkable performance of our proposed approach, which significantly reduces online interaction overhead while maintaining effective beam selection across diverse conditions including strict false alarm control, low signal-to-noise ratios, and high target velocities.