# Two-Timescale Synchronization and Migration for Digital Twin Networks: A Multi-Agent Deep Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2409.01092 · HTML: https://mlsanity.com/paper/2409.01092/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Digital-Twin-Enabled 6G: Vision, Architectural Trends, and Future Directions](https://mlsanity.com/paper/2102.12169/) (2021)
- [Path Planning for UAV-Mounted Mobile Edge Computing with Deep Reinforcement Learning](https://mlsanity.com/paper/2001.10268/) (2020)
- [A Survey on Digital Twins: Architecture, Enabling Technologies, Security and Privacy, and Future Prospects](https://mlsanity.com/paper/2301.13350/) (2023)
- [Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks](https://mlsanity.com/paper/2011.08430/) (2020)
- [Accuracy-Guaranteed Collaborative DNN Inference in Industrial IoT via Deep Reinforcement Learning](https://mlsanity.com/paper/2301.00130/) (2022)
- [Graph-Embedded Multi-Agent Learning for Smart Reconfigurable THz MIMO-NOMA Networks](https://mlsanity.com/paper/2107.07198/) (2021)
- [Bi-directional Digital Twin and Edge Computing in the Metaverse](https://mlsanity.com/paper/2211.08700/) (2022)
- [UAV-Assisted Intelligent Reflecting Surface Symbiotic Radio System](https://mlsanity.com/paper/2007.14029/) (2020)
- [Energy Efficient Computation Offloading in Aerial Edge Networks With Multi-Agent Cooperation](https://mlsanity.com/paper/2302.09021/) (2023)

## Built on by (most cited first, 3 of 3)

- [Distributed Coordination for Heterogeneous Non-Terrestrial Networks](https://mlsanity.com/paper/2502.17366/) (2025)
- [When Digital Twin Meets 6G: Concepts, Obstacles, and Research Prospects](https://mlsanity.com/paper/2409.02008/) (2024)
- [Energy-Efficient Federated Learning and Migration in Digital Twin Edge Networks](https://mlsanity.com/paper/2503.15822/) (2025)

## Abstract (arXiv)

Digital twins (DTs) have emerged as a promising enabler for representing the real-time states of physical worlds and realizing self-sustaining systems. In practice, DTs of physical devices, such as mobile users (MUs), are commonly deployed in multi-access edge computing (MEC) networks for the sake of reducing latency. To ensure the accuracy and fidelity of DTs, it is essential for MUs to regularly synchronize their status with their DTs. However, MU mobility introduces significant challenges to DT synchronization. Firstly, MU mobility triggers DT migration which could cause synchronization failures. Secondly, MUs require frequent synchronization with their DTs to ensure DT fidelity. Nonetheless, DT migration among MEC servers, caused by MU mobility, may occur infrequently. Accordingly, we propose a two-timescale DT synchronization and migration framework with reliability consideration by establishing a non-convex stochastic problem to minimize the long-term average energy consumption of MUs. We use Lyapunov theory to convert the reliability constraints and reformulate the new problem as a partially observable Markov decision-making process (POMDP). Furthermore, we develop a heterogeneous agent proximal policy optimization with Beta distribution (Beta-HAPPO) method to solve it. Numerical results show that our proposed Beta-HAPPO method achieves significant improvements in energy savings when compared with other benchmarks.