mlsanityData to 14 Oct 2026

Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare

arXiv 2305.01738 ↗First posted 2 May 2023cs.LG cs.AIcited by 42

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Many reinforcement learning (RL) applications have combinatorial action spaces, where each action is a composition of sub-actions. A standard RL approach ignores this inherent factorization structure, resulting in a potential failure to make meaningful inferences about rarely observed sub-action combinations; this is particularly problematic for offline settings, where data may be limited. In this work, we propose a form of linear Q-function decomposition induced by factored action spaces. We study the theoretical properties of our approach, identifying scenarios where it is guaranteed to lead to zero bias when used to approximate the Q-function. Outside the regimes with theoretical guarantees, we show that our approach can still be useful because it leads to better sample efficiency without necessarily sacrificing policy optimality, allowing us to achieve a better bias-variance trade-off. Across several offline RL problems using simulators and real-world datasets motivated by healthcare, we demonstrate that incorporating factored action spaces into value-based RL can result in better-performing policies. Our approach can help an agent make more accurate inferences within underexplored regions of the state-action space when applying RL to observational datasets.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 85 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

35 of 85 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  2. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  3. 2020Batch Value-function Approximation with Only Realizability11 Aug
  4. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  4. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  6. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  7. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 184
  8. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  9. 2016Off-policy evaluation for slate recommendationcited by 103
  10. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
  11. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  12. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
17 more
  1. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 58
  2. 2021Causally motivated Shortcut Removal Using Auxiliary Labelscited by 55
  3. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
  4. 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
  5. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  6. 2014Near-optimal Reinforcement Learning in Factored MDPscited by 41
  7. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  8. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  9. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  10. 2020Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorizationcited by 26
  11. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  12. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 23
  13. 2018Factorized Q-Learning for Large-Scale Multi-Agent Systemscited by 19
  14. 2020On the Rademacher Complexity of Linear Hypothesis Setscited by 17
  15. 2017Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learningcited by 15
  16. 2021Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPscited by 8
  17. 2021Causal Markov Decision Processes: Learning Good Interventions Efficientlycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 18; 2025: 10; 2026: 6.

42 papers.

Sort:

By area:

  1. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  2. 2023Reliable Off-Policy Learning for Dosage Combinationscited by 15
  3. 2024A Review of Safe Reinforcement Learning Methods for Modern Power Systemscited by 11
  4. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  5. 2024DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regimecited by 8
  6. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  7. 2023Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluationcited by 7
  8. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  9. 2024Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstractioncited by 6
  10. 2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
  11. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 6
  12. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
30 more
  1. 2023Dynamic Neighborhood Construction for Structured Large Discrete Action Spacescited by 4
  2. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learningcited by 4
  3. 2025Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatmentcited by 4
  4. 2023Leveraging Factored Action Spaces for Off-Policy Evaluationcited by 3
  5. 2024BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spacescited by 3
  6. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  7. 2025Advancing Safe Mechanical Ventilation Using Offline RL With Hybrid Actions and Clinically Aligned Rewardscited by 3
  8. 2025Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AIcited by 3
  9. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learningcited by 3
  10. 2023Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Carecited by 2
  11. 2024How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Modelscited by 2
  12. 2024SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systemscited by 2
  13. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatmentcited by 2
  14. 2023Offline Reinforcement Learning with Additional Covering Distributionscited by 1
  15. 2024Offline Trajectory Optimization for Offline Reinforcement Learningcited by 1
  16. 2024State-Constrained Offline Reinforcement Learningcited by 1
  17. 2024From Basic to Extra Features: Hypergraph Transformer Pretrain-then-Finetuning for Balanced Clinical Predictions on EHRcited by 1
  18. 2025ExOSITO: Explainable Off-Policy Learning with Side Information for Intensive Care Unit Blood Test Orderscited by 1
  19. 2025Diffusion Policies with Offline and Inverse Reinforcement Learning for Promoting Physical Activity in Older Adults Using Wearable Sensorscited by 1
  20. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learningcited by 0
  21. 2024Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectoriescited by 0
  22. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learningcited by 0
  23. 2025Q-function Decomposition with Intervention Semantics with Factored Action Spacescited by 0
  24. 2025SAINT: Attention-Based Policies for Discrete Combinatorial Action Spacescited by 0
  25. 2026Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initializationcited by 0
  26. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0
  27. 2026MARLIN: Multi-Agent Reinforcement Learning for Incremental DAG Discoverycited by 0
  28. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimationcited by 0
  29. 2026Learning from the Test: Self-Referential Differential Testing for Deep RL Agentscited by 0
  30. 2026Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  2. 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
  3. 2024Learning in complex action spaces without policy gradients
  4. 2026Finite-Sample Theory for Fitted Q-Iteration When Actions Are Functions
  5. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
  6. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
  7. 2022Offline Policy Evaluation and Optimization under Confounding
  8. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learning