# RLOps: Development Life-cycle of Reinforcement Learning Aided Open RAN

arXiv: https://arxiv.org/abs/2111.06978 · HTML: https://mlsanity.com/paper/2111.06978/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 97 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Builds on (most cited first, 30 of 39)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Robust Deep Reinforcement Learning with Adversarial Attacks](https://mlsanity.com/paper/1712.03632/) (2017)
- [AliGraph: A Comprehensive Graph Neural Network Platform](https://mlsanity.com/paper/1902.08730/) (2019)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspective](https://mlsanity.com/paper/1811.03402/) (2018)
- [Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1906.04737/) (2019)
- [Deep Learning-Based Channel Estimation](https://mlsanity.com/paper/1810.05893/) (2018)
- [A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments](https://mlsanity.com/paper/2005.10619/) (2020)
- [Horizon: Facebook's Open Source Applied Reinforcement Learning Platform](https://mlsanity.com/paper/1811.00260/) (2018)
- [Intelligence and Learning in O-RAN for Data-driven NextG Cellular Networks](https://mlsanity.com/paper/2012.01263/) (2020)
- [Evaluating the Performance of Reinforcement Learning Algorithms](https://mlsanity.com/paper/2006.16958/) (2020)
- [Learning Curriculum Policies for Reinforcement Learning](https://mlsanity.com/paper/1812.00285/) (2018)
- [Learning with Safety Constraints: Sample Complexity of Reinforcement Learning for Constrained MDPs](https://mlsanity.com/paper/2008.00311/) (2020)
- [A Data Quality-Driven View of MLOps](https://mlsanity.com/paper/2102.07750/) (2021)
- [OFDM-Autoencoder for End-to-End Learning of Communications Systems](https://mlsanity.com/paper/1803.05815/) (2018)
- [Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning](https://mlsanity.com/paper/1805.06146/) (2018)

## Built on by (most cited first, 15 of 15)

- [Safe and Accelerated Deep Reinforcement Learning-based O-RAN Slicing: A Hybrid Transfer Learning Approach](https://mlsanity.com/paper/2309.07265/) (2023)
- [ML-Enabled Open RAN: A Comprehensive Survey of Architectures, Challenges, and Opportunities](https://mlsanity.com/paper/2604.01239/) (2026)
- [Sim2real for Reinforcement Learning Driven Next Generation Networks](https://mlsanity.com/paper/2206.03846/) (2022)
- [Transmit Power Control for Indoor Small Cells: A Method Based on Federated Reinforcement Learning](https://mlsanity.com/paper/2209.13536/) (2022)
- [Intelligent Closed-loop RAN Control with xApps in OpenRAN Gym](https://mlsanity.com/paper/2208.14877/) (2022)
- [Adapting MLOps for Diverse In-Network Intelligence in 6G Era: Challenges and Solutions](https://mlsanity.com/paper/2410.18793/) (2024)
- [An Intentional Forgetting-Driven Self-Healing Method For Deep Reinforcement Learning Systems](https://mlsanity.com/paper/2308.12445/) (2023)
- [Past, Present, Future: A Comprehensive Exploration of AI Use Cases in the UMBRELLA IoT Testbed](https://mlsanity.com/paper/2401.13346/) (2024)
- [Building the Self-Improvement Loop: Error Detection and Correction in Goal-Oriented Semantic Communications](https://mlsanity.com/paper/2411.01544/) (2024)
- [Variational Autoencoder Assisted Neural Network Likelihood RSRP Prediction Model](https://mlsanity.com/paper/2207.00166/) (2022)
- [FORLORN: A Framework for Comparing Offline Methods and Reinforcement Learning for Optimization of RAN Parameters](https://mlsanity.com/paper/2209.13540/) (2022)
- [Sparsity-Aware Intelligent Massive Random Access Control in Open RAN: A Reinforcement Learning Based Approach](https://mlsanity.com/paper/2303.02657/) (2023)
- [Demo: A Digital Twin of the 5G Radio Access Network for Anomaly Detection Functionality](https://mlsanity.com/paper/2308.15973/) (2023)
- [Federated Neuroevolution O-RAN: Enhancing the Robustness of Deep Reinforcement Learning xApps](https://mlsanity.com/paper/2506.12812/) (2025)
- [Latency-aware Human-in-the-Loop Reinforcement Learning for Semantic Communications](https://mlsanity.com/paper/2602.15640/) (2026)

## Abstract (arXiv)

Radio access network (RAN) technologies continue to evolve, with Open RAN gaining the most recent momentum. In the O-RAN specifications, the RAN intelligent controllers (RICs) are software-defined orchestration and automation functions for the intelligent management of RAN. This article introduces principles for machine learning (ML), in particular, reinforcement learning (RL) applications in the O-RAN stack. Furthermore, we review the state-of-the-art research in wireless networks and cast it onto the RAN framework and the hierarchy of the O-RAN architecture. We provide a taxonomy for the challenges faced by ML/RL models throughout the development life-cycle: from the system specification to production deployment (data acquisition, model design, testing and management, etc.). To address the challenges, we integrate a set of existing MLOps principles with unique characteristics when RL agents are considered. This paper discusses a systematic model development, testing and validation life-cycle, termed: RLOps. We discuss fundamental parts of RLOps, which include: model specification, development, production environment serving, operations monitoring and safety/security. Based on these principles, we propose the best practices for RLOps to achieve an automated and reproducible model development process. At last, a holistic data analytics platform rooted in the O-RAN deployment is designed and implemented, aiming to embrace and fulfil the aforementioned principles and best practices of RLOps.