# Optimizing Reinforcement Learning Training over Digital Twin Enabled Multi-fidelity Networks

arXiv: https://arxiv.org/abs/2603.08931 · HTML: https://mlsanity.com/paper/2603.08931/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [A Vision of 6G Wireless Systems: Applications, Trends, Technologies, and Open Research Problems](https://mlsanity.com/paper/1902.10265/) (2019)
- [Robust Deep Reinforcement Learning through Adversarial Loss](https://mlsanity.com/paper/2008.01976/) (2020)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagation](https://mlsanity.com/paper/2210.07394/) (2022)
- [Digital Twin of Wireless Systems: Overview, Taxonomy, Challenges, and Opportunities](https://mlsanity.com/paper/2202.02559/) (2022)
- [Wireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both?](https://mlsanity.com/paper/1902.02647/) (2019)
- [Accelerating DNN Training in Wireless Federated Edge Learning Systems](https://mlsanity.com/paper/1905.09712/) (2019)
- [Real-Time Digital Twins: Vision and Research Directions for 6G and Beyond](https://mlsanity.com/paper/2301.11283/) (2023)
- [Wireless Network Digital Twin for 6G: Generative AI as A Key Enabler](https://mlsanity.com/paper/2311.17451/) (2023)
- [Deep Learning for Wireless Communications: An Emerging Interdisciplinary Paradigm](https://mlsanity.com/paper/2007.05952/) (2020)
- [Online Antenna Tuning in Heterogeneous Cellular Networks with Deep Reinforcement Learning](https://mlsanity.com/paper/1903.06787/) (2019)
- [Edge Continual Learning for Dynamic Digital Twins over Wireless Networks](https://mlsanity.com/paper/2204.04795/) (2022)
- [Digital Twin Based Beam Prediction: Can we Train in the Digital World and Deploy in Reality?](https://mlsanity.com/paper/2301.07682/) (2023)
- [Digital Twin for Networking: A Data-driven Performance Modeling Perspective](https://mlsanity.com/paper/2206.00310/) (2022)
- [Robust Bayesian Learning for Reliable Wireless AI: Framework and Applications](https://mlsanity.com/paper/2207.00300/) (2022)
- [Optimization of Image Transmission in a Cooperative Semantic Communication Networks](https://mlsanity.com/paper/2301.00433/) (2023)
- [Coordinated Reinforcement Learning for Optimizing Mobile Networks](https://mlsanity.com/paper/2109.15175/) (2021)
- [Uncertainty Injection: A Deep Learning Method for Robust Optimization](https://mlsanity.com/paper/2302.12304/) (2023)
- [Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network](https://mlsanity.com/paper/2310.09299/) (2023)
- [Optimizing Communication and Device Clustering for Clustered Federated Learning with Differential Privacy](https://mlsanity.com/paper/2507.07320/) (2025)
- [Optimizing Wireless Resource Management and Synchronization in Digital Twin Networks](https://mlsanity.com/paper/2502.05116/) (2025)
- [Optimizing Model Splitting and Device Task Assignment for Deceptive Signal Assisted Private Multi-hop Split Learning](https://mlsanity.com/paper/2507.07323/) (2025)
- [Optimal Base Station Antenna Downtilt in Downlink Cellular Networks](https://mlsanity.com/paper/1802.07479/) (2018)
- [Continual Reinforcement Learning for Digital Twin Synchronization Optimization](https://mlsanity.com/paper/2501.08045/) (2025)

## Abstract (arXiv)

In this paper, we investigate a novel digital network twin (DNT) assisted deep learning (DL) model training framework. In particular, we consider a physical network where a base station (BS) uses several antennas to serve multiple mobile users, and a DNT that is a virtual representation of the physical network. The BS must adjust its antenna tilt angles to optimize the data rates of all users. Due to user mobility, the BS may not be able to accurately track network dynamics such as wireless channels and user mobilities. Hence, a reinforcement learning (RL) approach is used to dynamically adjust the antenna tilt angles. To train the RL, we can use data collected from the physical network and the DNT. The data collected from the physical network is more accurate but incurs more communication overhead compared to the data collected from the DNT. Therefore, it is necessary to determine the ratio of data collected from the physical network and the DNT to improve the training of the RL model. We formulate this problem as an optimization problem whose goal is to jointly optimize the tilt angle adjustment policy and the data collection strategy, aiming to maximize the data rates of all users while constraining the time delay introduced by collecting data from the physical network. To solve this problem, we propose a hierarchical RL framework that integrates robust adversarial loss and proximal policy optimization (PPO). Simulation results show that our proposed method reduces the physical network data collection delay by up to 28.01% and 1x compared to a hierarchical RL that uses vanilla PPO as the first level RL, and the baseline that uses robust-RL at the first level and selects the data collection ratio randomly.