# Sim2real for Reinforcement Learning Driven Next Generation Networks

arXiv: https://arxiv.org/abs/2206.03846 · HTML: https://mlsanity.com/paper/2206.03846/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 9 of 9)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real](https://mlsanity.com/paper/2006.09001/) (2020)
- [An empirical investigation of the challenges of real-world reinforcement learning](https://mlsanity.com/paper/2003.11881/) (2020)
- [Intelligence and Learning in O-RAN for Data-driven NextG Cellular Networks](https://mlsanity.com/paper/2012.01263/) (2020)
- [Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertainty](https://mlsanity.com/paper/2010.04870/) (2020)
- [RLOps: Development Life-cycle of Reinforcement Learning Aided Open RAN](https://mlsanity.com/paper/2111.06978/) (2021)

## Built on by (most cited first, 3 of 3)

- [Transmit Power Control for Indoor Small Cells: A Method Based on Federated Reinforcement Learning](https://mlsanity.com/paper/2209.13536/) (2022)
- [Towards Practical Operation of Deep Reinforcement Learning Agents in Real-World Network Management at Open RAN Edges](https://mlsanity.com/paper/2410.23086/) (2024)
- [The Role and Applications of Airport Digital Twin in Cyberattack Protection during the Generative AI Era](https://mlsanity.com/paper/2408.05248/) (2024)

## Abstract (arXiv)

The next generation of networks will actively embrace artificial intelligence (AI) and machine learning (ML) technologies for automation networks and optimal network operation strategies. The emerging network structure represented by Open RAN (O-RAN) conforms to this trend, and the radio intelligent controller (RIC) at the centre of its specification serves as an ML applications host. Various ML models, especially Reinforcement Learning (RL) models, are regarded as the key to solving RAN-related multi-objective optimization problems. However, it should be recognized that most of the current RL successes are confined to abstract and simplified simulation environments, which may not directly translate to high performance in complex real environments. One of the main reasons is the modelling gap between the simulation and the real environment, which could make the RL agent trained by simulation ill-equipped for the real environment. This issue is termed as the sim2real gap. This article brings to the fore the sim2real challenge within the context of O-RAN. Specifically, it emphasizes the characteristics, and benefits that the digital twins (DT) could have as a place for model development and verification. Several use cases are presented to exemplify and demonstrate failure modes of the simulations trained RL model in real environments. The effectiveness of DT in assisting the development of RL algorithms is discussed. Then the current state of the art learning-based methods commonly used to overcome the sim2real challenge are presented. Finally, the development and deployment concerns for the RL applications realisation in O-RAN are discussed from the view of the potential issues like data interaction, environment bottlenecks, and algorithm design.