mlsanityData to 14 Oct 2026

Q-function Decomposition with Intervention Semantics with Factored Action Spaces

arXiv 2504.21326 ↗First posted 30 Apr 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Many practical reinforcement learning environments have a discrete factored action space that induces a large combinatorial set of actions, thereby posing significant challenges. Existing approaches leverage the regular structure of the action space and resort to a linear decomposition of Q-functions, which avoids enumerating all combinations of factored actions. In this paper, we consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions using causal effect estimation from the no unobserved confounder setting in causal statistics. This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms. The proposed approach is shown to improve sample complexity in a model-based reinforcement learning setting. We demonstrate improvements in sample efficiency compared to state-of-the-art baselines in online continuous control environments and a real-world offline sepsis treatment environment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 57 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

28 of 57 references.

References · 28

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2018DeepMind Control Suitecited by 956
  5. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  6. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  7. 2016Learning Representations for Counterfactual Inferencecited by 329
  8. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  9. 2019Causality for Machine Learningcited by 243
  10. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  11. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  12. 2022Weakly supervised causal representation learningcited by 140
16 more
  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  3. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
  4. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 70
  5. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  6. 2022Causal Inference in Recommender Systems: A Survey and Future Directionscited by 44
  7. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  8. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
  9. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  10. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  11. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  12. 2022Solving Continuous Control via Q-learningcited by 24
  13. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 23
  14. 2023Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluationcited by 7
  15. 2023Leveraging Factored Action Spaces for Off-Policy Evaluationcited by 3
  16. 2024Why Online Reinforcement Learning is Causalcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
  2. 2026Finite-Sample Theory for Fitted Q-Iteration When Actions Are Functions
  3. 2024Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition
  4. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
  5. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  6. 2022Learning Value Functions from Undirected State-only Experience
  7. 2022Causal Deep Reinforcement Learning Using Observational Data
  8. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders