# TwinLoop: Simulation-in-the-Loop Digital Twins for Online Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2604.06610 · HTML: https://mlsanity.com/paper/2604.06610/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments](https://mlsanity.com/paper/2005.10619/) (2020)
- [Adaptive Federated Learning and Digital Twin for Industrial Internet of Things](https://mlsanity.com/paper/2010.13058/) (2020)
- [Digital Twin-Assisted Efficient Reinforcement Learning for Edge Task Scheduling](https://mlsanity.com/paper/2208.01781/) (2022)
- [Feature-Model-Guided Online Learning for Self-Adaptive Systems](https://mlsanity.com/paper/1907.09158/) (2019)
- [Applying Machine Learning in Self-Adaptive Systems: A Systematic Literature Review](https://mlsanity.com/paper/2103.04112/) (2021)
- [Decentralized Self-Adaptive Systems: A Mapping Study](https://mlsanity.com/paper/2103.09074/) (2021)
- [Large Language Models for Explainable Decisions in Dynamic Digital Twins](https://mlsanity.com/paper/2405.14411/) (2024)
- [Knowledge Equivalence in Digital Twins of Intelligent Systems](https://mlsanity.com/paper/2204.07481/) (2022)
- [Explainable Human-in-the-loop Dynamic Data-Driven Digital Twins](https://mlsanity.com/paper/2207.09106/) (2022)
- [Digital Twin Vehicular Edge Computing Network: Task Offloading and Resource Allocation](https://mlsanity.com/paper/2407.11310/) (2024)
- [Dynamic Data-Driven Digital Twins for Blockchain Systems](https://mlsanity.com/paper/2312.04226/) (2023)
- [Intelligent Offloading in Vehicular Edge Computing: A Comprehensive Review of Deep Reinforcement Learning Approaches and Architectures](https://mlsanity.com/paper/2502.06963/) (2025)
- [A Digital Twin-based Multi-Agent Reinforcement Learning Framework for Vehicle-to-Grid Coordination](https://mlsanity.com/paper/2510.27289/) (2025)

## Abstract (arXiv)

Decentralised online learning enables runtime adaptation in cyber-physical multi-agent systems, but when operating conditions change, learned policies often require substantial trial-and-error interaction before recovering performance. To address this, we propose TwinLoop, a simulation-in-the-loop digital twin framework for online multi-agent reinforcement learning. When a context shift occurs, the digital twin is triggered to reconstruct the current system state, initialise from the latest agent policies, and perform accelerated policy improvement with simulation what-if analysis before synchronising updated parameters back to the agents in the physical system. We evaluate TwinLoop in a vehicular edge computing task-offloading scenario with changing workload and infrastructure conditions. The results suggest that digital twins can improve post-shift adaptation efficiency and reduce reliance on costly online trial-and-error.