# Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control

arXiv: https://arxiv.org/abs/2603.03932 · HTML: https://mlsanity.com/paper/2603.03932/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- Offline RL and imitation learning from data / [Offline multi-agent RL](https://mlsanity.com/offlinerl/offline-multi-agent-rl/)

## Builds on (most cited first, 19 of 19)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL](https://mlsanity.com/paper/2209.03993/) (2022)
- [d3rlpy: An Offline Deep Reinforcement Learning Library](https://mlsanity.com/paper/2111.03788/) (2021)
- [You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments](https://mlsanity.com/paper/2205.15967/) (2022)
- [A Review of Uncertainty for Deep Reinforcement Learning](https://mlsanity.com/paper/2208.09052/) (2022)
- [Critic-Guided Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2312.13716/) (2023)
- [ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning](https://mlsanity.com/paper/2309.05915/) (2023)
- [When should we prefer Decision Transformers for Offline Reinforcement Learning?](https://mlsanity.com/paper/2305.14550/) (2023)
- [Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling](https://mlsanity.com/paper/2406.00079/) (2024)
- [Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets](https://mlsanity.com/paper/2311.11423/) (2023)
- [Advancing RAN Slicing with Offline Reinforcement Learning](https://mlsanity.com/paper/2312.10547/) (2023)
- [Offline Reinforcement Learning and Sequence Modeling for Downlink Link Adaptation](https://mlsanity.com/paper/2410.23031/) (2024)
- [Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?](https://mlsanity.com/paper/2507.10174/) (2025)
- [Offline Reinforcement Learning for Mobility Robustness Optimization](https://mlsanity.com/paper/2506.22793/) (2025)

## Abstract (arXiv)

Offline Reinforcement Learning (RL) is a promising approach for next-generation wireless networks, where online exploration is unsafe and large amounts of operational data can be reused across the model lifecycle. However, the behavior of offline RL algorithms under genuinely stochastic dynamics -- inherent to wireless systems due to fading, noise, and traffic mobility -- remains insufficiently understood. We address this gap by evaluating Bellman-based (Conservative Q-Learning), sequence-based (Decision Transformers), and hybrid (Critic-Guided Decision Transformers) offline RL methods in an open-access stochastic telecom environment (mobile-env). Our results show that Conservative Q-Learning consistently produces more robust policies across different sources of stochasticity, making it a reliable default choice in lifecycle-driven AI management frameworks. Sequence-based methods remain competitive and can outperform Bellman-based approaches when sufficient high-return trajectories are available. These findings provide practical guidance for offline RL algorithm selection in AI-driven network control pipelines, such as O-RAN and future 6G functions, where robustness and data availability are key operational constraints.