# Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets

arXiv: https://arxiv.org/abs/2311.11423 · HTML: https://mlsanity.com/paper/2311.11423/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 19 Nov 2023.

## Placed in

- Offline RL and imitation learning from data / [Offline multi-agent RL](https://mlsanity.com/offlinerl/offline-multi-agent-rl/)

## Builds on (most cited first, 12 of 12)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism](https://mlsanity.com/paper/2103.12021/) (2021)
- [Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning](https://mlsanity.com/paper/2106.04895/) (2021)
- [When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?](https://mlsanity.com/paper/2204.05618/) (2022)
- [Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks](https://mlsanity.com/paper/1808.00490/) (2018)
- [Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2002.06215/) (2020)
- [A Deep Q-Learning Method for Downlink Power Allocation in Multi-Cell Networks](https://mlsanity.com/paper/1904.13032/) (2019)
- [Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints](https://mlsanity.com/paper/2306.06265/) (2023)

## Built on by (most cited first, 12 of 12)

- [Adaptive Layer Splitting for Wireless LLM Inference in Edge Computing: A Model-Based Reinforcement Learning Approach](https://mlsanity.com/paper/2406.02616/) (2024)
- [Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2402.08421/) (2024)
- [Advancing RAN Slicing with Offline Reinforcement Learning](https://mlsanity.com/paper/2312.10547/) (2023)
- [An Offline Multi-Agent Reinforcement Learning Framework for Radio Resource Management](https://mlsanity.com/paper/2501.12991/) (2025)
- [Offline and Distributional Reinforcement Learning for Radio Resource Management](https://mlsanity.com/paper/2409.16764/) (2024)
- [Average Reward Reinforcement Learning for Wireless Radio Resource Management](https://mlsanity.com/paper/2501.06700/) (2025)
- [Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis](https://mlsanity.com/paper/2505.13768/) (2025)
- [NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation](https://mlsanity.com/paper/2411.04138/) (2024)
- [Offline Diffusion Policy for Multi-User Delay-Constrained Scheduling](https://mlsanity.com/paper/2501.12942/) (2025)
- [Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control](https://mlsanity.com/paper/2603.03932/) (2026)
- [Diffusion Offline Reinforcement Learning for Fair and Energy-Efficient UAV-Assisted Wireless Networks](https://mlsanity.com/paper/2606.16331/) (2026)
- [Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management](https://mlsanity.com/paper/2608.28878/) (2026)

## Abstract (arXiv)

The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, which may be undesirable given the potential performance loss due to the unavoidable exploration in RL. In this work, we first investigate the use of offline RL algorithms in solving the RRM problem. We evaluate several state-of-the-art offline RL algorithms, including behavior constrained Q-learning (BCQ), conservative Q-learning (CQL), and implicit Q-learning (IQL), for a specific RRM problem that aims at maximizing a linear combination of sum and 5-percentile rates via user scheduling. We observe that the performance of offline RL for the RRM problem depends critically on the behavior policy used for data collection, and further propose a novel offline RL solution that leverages heterogeneous datasets collected by different behavior policies. We show that with a proper mixture of the datasets, offline RL can produce a near-optimal RL policy even when all involved behavior policies are highly suboptimal.