mlsanityData to 14 Oct 2026

An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces

arXiv 2411.11088 ↗First posted 17 Nov 2024stat.ML cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poses substantial challenges or risks. Pivotal to the success of transferring RL offline is mitigating overestimation bias in value estimates for state-action pairs absent from data. Whilst numerous approaches have been proposed in recent years, these tend to focus primarily on continuous or small-scale discrete action spaces. Factorised discrete action spaces, on the other hand, have received relatively little attention, despite many real-world problems naturally having factorisable actions. In this work, we undertake a formative investigation into offline reinforcement learning in factorisable action spaces. Using value-decomposition as formulated in DecQN as a foundation, we present the case for a factorised approach and conduct an extensive empirical evaluation of several offline techniques adapted to the factorised setting. In the absence of established benchmarks, we introduce a suite of our own comprising datasets of varying quality and task complexity. Advocating for reproducible research and innovation, we make all datasets available for public use alongside our code base.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 68 references that are in the map. First posted on arXiv 17 Nov 2024.

Builds on

54 of 68 references.

Closest ancestors · 21

Same direction, newest first

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  2. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning26 Mar
  3. 2023Anti-Exploration by Random Network Distillation31 Jan
  4. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  5. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021Offline RL Without Off-Policy Evaluation16 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
9 more
  1. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  2. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  9. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 33

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  9. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  10. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  12. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
21 more
  1. 2018Distributed Prioritized Experience Replaycited by 269
  2. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  3. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  4. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  5. 2020Model-Based Offline Planningcited by 129
  6. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  7. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
  8. 2021Learning and Planning in Complex Action Spacescited by 77
  9. 2019Learning Action Representations for Reinforcement Learningcited by 75
  10. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  11. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 65
  12. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  13. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
  14. 2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 38
  15. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 36
  16. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  17. 2019Growing Action Spacescited by 29
  18. 2022Solving Continuous Control via Q-learningcited by 24
  19. 2017Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learningcited by 15
  20. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  21. 2023Leveraging Factored Action Spaces for Off-Policy Evaluationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025SAINT: Attention-Based Policies for Discrete Combinatorial Action Spacescited by 0
  2. 2026Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initializationcited by 0
  3. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning Hands-On
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2024BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces
  4. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  5. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2021Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
  8. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model