# A Bayesian Framework of Deep Reinforcement Learning for Joint O-RAN/MEC Orchestration

arXiv: https://arxiv.org/abs/2312.16142 · HTML: https://mlsanity.com/paper/2312.16142/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 26 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://mlsanity.com/paper/1802.09127/) (2018)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Efficient Exploration through Bayesian Deep Q-Networks](https://mlsanity.com/paper/1802.04412/) (2018)
- [Intelligence and Learning in O-RAN for Data-driven NextG Cellular Networks](https://mlsanity.com/paper/2012.01263/) (2020)
- [ColO-RAN: Developing Machine Learning-based xApps for Open RAN Closed-loop Control on Programmable Experimental Platforms](https://mlsanity.com/paper/2112.09559/) (2021)
- [srsLTE: An Open-Source Platform for LTE Evolution and Experimentation](https://mlsanity.com/paper/1602.04629/) (2016)
- [Open, Programmable, and Virtualized 5G Networks: State-of-the-Art and the Road Ahead](https://mlsanity.com/paper/2005.10027/) (2020)
- [Vehicular Cooperative Perception Through Action Branching and Federated Reinforcement Learning](https://mlsanity.com/paper/2012.03414/) (2020)
- [Reinforcement Learning Based Dynamic Function Splitting in Disaggregated Green Open RANs](https://mlsanity.com/paper/2012.03213/) (2020)
- [Sl-EDGE: Network Slicing at the Edge](https://mlsanity.com/paper/2005.00886/) (2020)
- [Adaptive Resource Allocation for Virtualized Base Stations in O-RAN with Online Learning](https://mlsanity.com/paper/2309.01730/) (2023)
- [The Complexity-Rate Tradeoff of Centralized Radio Access Networks](https://mlsanity.com/paper/1503.08585/) (2015)
- [Learning-Based Orchestration for Dynamic Functional Split and Resource Allocation in vRANs](https://mlsanity.com/paper/2205.07518/) (2022)
- [Deep Reinforcement Learning for Orchestrating Cost-Aware Reconfigurations of vRANs](https://mlsanity.com/paper/2208.05282/) (2022)

## Abstract (arXiv)

Multi-access Edge Computing (MEC) can be implemented together with Open Radio Access Network (O-RAN) over commodity platforms to offer low-cost deployment and bring the services closer to end-users. In this paper, a joint O-RAN/MEC orchestration using a Bayesian deep reinforcement learning (RL)-based framework is proposed that jointly controls the O-RAN functional splits, the allocated resources and hosting locations of the O-RAN/MEC services across geo-distributed platforms, and the routing for each O-RAN/MEC data flow. The goal is to minimize the long-term overall network operation cost and maximize the MEC performance criterion while adapting possibly time-varying O-RAN/MEC demands and resource availability. This orchestration problem is formulated as Markov decision process (MDP). However, the system consists of multiple BSs that share the same resources and serve heterogeneous demands, where their parameters have non-trivial relations. Consequently, finding the exact model of the underlying system is impractical, and the formulated MDP renders in a large state space with multi-dimensional discrete action. To address such modeling and dimensionality issues, a novel model-free RL agent is proposed for our solution framework. The agent is built from Double Deep Q-network (DDQN) that tackles the large state space and is then incorporated with action branching, an action decomposition method that effectively addresses the multi-dimensional discrete action with linear increase complexity. Further, an efficient exploration-exploitation strategy under a Bayesian framework using Thomson sampling is proposed to improve the learning performance and expedite its convergence. Trace-driven simulations are performed using an O-RAN-compliant model. The results show that our approach is data-efficient (i.e., converges faster) and increases the returned reward by 32% than its non-Bayesian version.