mlsanityData to 7 Oct 2026

Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Critic

arXiv 2506.05445 ↗First posted 5 Jun 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Hidden confounders that influence both states and actions can bias policy learning in reinforcement learning (RL), leading to suboptimal or non-generalizable behavior. Most RL algorithms ignore this issue, learning policies from observational trajectories based solely on statistical associations rather than causal effects. We propose DoSAC (Do-Calculus Soft Actor-Critic with Backdoor Adjustment), a principled extension of the SAC algorithm that corrects for hidden confounding via causal intervention estimation. DoSAC estimates the interventional policy π(a | do(s)) using the backdoor criterion, without requiring access to true confounders or causal labels. To achieve this, we introduce a learnable Backdoor Reconstructor that infers pseudo-past variables (previous state and action) from the current state to enable backdoor adjustment from observational data. This module is integrated into a soft actor-critic framework to compute both the interventional policy and its entropy. Empirical results on continuous control benchmarks show that DoSAC outperforms baselines under confounded settings, with improved robustness, generalization, and policy reliability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 47 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

24 of 47 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016OpenAI Gymcited by 1,684
  6. 2019Causality for Machine Learningcited by 242
  7. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  8. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 61
  9. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  10. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 57
  11. 2020Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learningcited by 53
  12. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
12 more
  1. 2022Causal Imitation Learning with Unobserved Confounderscited by 40
  2. 2017Budgeted Experiment Design for Causal Structure Learningcited by 38
  3. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  4. 2022Causal Imitation Learning under Temporally Correlated Noisecited by 28
  5. 2022Sequential Causal Imitation Learning with Unobserved Confounderscited by 25
  6. 2021Federated Causal Inference in Heterogeneous Observational Datacited by 24
  7. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  8. 2023An Adaptive Kernel Approach to Federated Learning of Heterogeneous Causal Effectscited by 19
  9. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  10. 2022Robust Policy Optimization in Deep Reinforcement Learningcited by 14
  11. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  12. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5

Built on by

Not cited yet.