# Provable Performance Bounds for Digital Twin-driven Deep Reinforcement Learning in Wireless Networks: A Novel Digital-Twin Bisimulation Metric

arXiv: https://arxiv.org/abs/2502.17983 · HTML: https://mlsanity.com/paper/2502.17983/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 60 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 22 of 22)

- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial](https://mlsanity.com/paper/2011.03615/) (2020)
- [Edge Learning for B5G Networks with Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing](https://mlsanity.com/paper/2206.00422/) (2022)
- [Towards Robust Bisimulation Metric Learning](https://mlsanity.com/paper/2110.14096/) (2021)
- [Digital Twin of Wireless Systems: Overview, Taxonomy, Challenges, and Opportunities](https://mlsanity.com/paper/2202.02559/) (2022)
- [Metrics and continuity in reinforcement learning](https://mlsanity.com/paper/2102.01514/) (2021)
- [6G Digital Twin Networks: From Theory to Practice](https://mlsanity.com/paper/2212.02032/) (2022)
- [Wireless Network Digital Twin for 6G: Generative AI as A Key Enabler](https://mlsanity.com/paper/2311.17451/) (2023)
- [On the Generalization for Transfer Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2207.05377/) (2022)
- [Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges](https://mlsanity.com/paper/2406.07857/) (2024)
- [Attention-based QoE-aware Digital Twin Empowered Edge Computing for Immersive Virtual Reality](https://mlsanity.com/paper/2305.08569/) (2023)
- [Digital Twin-Enhanced Deep Reinforcement Learning for Resource Management in Networks Slicing](https://mlsanity.com/paper/2311.16876/) (2023)
- [Optimal and Fast Real-time Resources Slicing with Deep Dueling Neural Networks](https://mlsanity.com/paper/1902.09696/) (2019)
- [DQLEL: Deep Q-Learning for Energy-Optimized LoS/NLoS UWB Node Selection](https://mlsanity.com/paper/2108.13157/) (2021)
- [The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication](https://mlsanity.com/paper/2103.13026/) (2021)
- [Multiverse at the Edge: Interacting Real World and Digital Twins for Wireless Beamforming](https://mlsanity.com/paper/2305.10350/) (2023)
- [Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network](https://mlsanity.com/paper/2310.09299/) (2023)
- [Digital Twin Based User-Centric Resource Management for Multicast Short Video Streaming](https://mlsanity.com/paper/2308.08995/) (2023)
- [Leveraging Digital Cousins for Ensemble Q-Learning in Large-Scale Wireless Networks](https://mlsanity.com/paper/2402.08022/) (2024)
- [Large Vision Model-Enhanced Digital Twin with Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks](https://mlsanity.com/paper/2410.07611/) (2024)

## Built on by (most cited first, 1 of 1)

- [Learning to Gridize: Segment Physical World by Wireless Communication Channel](https://mlsanity.com/paper/2507.15386/) (2025)

## Abstract (arXiv)

Digital twin (DT)-driven deep reinforcement learning (DRL) has emerged as a promising paradigm for wireless network optimization, offering safe and efficient training environment for policy exploration. However, in theory existing methods cannot always guarantee real-world performance of DT-trained policies before actual deployment, due to the absence of a universal metric for assessing DT's ability to support reliable DRL training transferrable to physical networks. In this paper, we propose the DT bisimulation metric (DT-BSM), a novel metric based on the Wasserstein distance, to quantify the discrepancy between Markov decision processes (MDPs) in both the DT and the corresponding real-world wireless network environment. We prove that for any DT-trained policy, the sub-optimality of its performance (regret) in the real-world deployment is bounded by a weighted sum of the DT-BSM and its sub-optimality within the MDP in the DT. Then, a modified DT-BSM based on the total variation distance is also introduced to avoid the prohibitive calculation complexity of Wasserstein distance for large-scale wireless network scenarios. Further, to tackle the challenge of obtaining accurate transition probabilities of the MDP in real world for the DT-BSM calculation, we propose an empirical DT-BSM method based on statistical sampling. We prove that the empirical DT-BSM always converges to the desired theoretical one, and quantitatively establish the relationship between the required sample size and the target level of approximation accuracy. Numerical experiments validate this first theoretical finding on the provable and calculable performance bounds for DT-driven DRL.