mlsanityData to 7 Oct 2026

Counterfactually Safe Reinforcement Learning

arXiv 2605.25114 ↗First posted 24 May 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning algorithms are generally designed to maximize the expected return across a population. However, a policy that is optimal on average may be suboptimal for certain individuals, leading to potential safety concerns. To address this, we first formalize the notion of individual harm from a counterfactual perspective and define harm as the event in which a chosen action results in a strictly worse outcome than a baseline alternative. We then propose a general two-stage procedure for learning policies that maximize the expected return while accounting for individual harm. We further establish the finite-sample properties of the learned policy, derive an upper bound on its sub-optimality gap, and show that the harm rate remains well-controlled. Numerical experiments on both simulated and real-world datasets demonstrate the effectiveness of the proposed approach.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 83 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

39 of 83 references.

References · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2017Counterfactual Fairnesscited by 802
  5. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  6. 2017Robust Adversarial Reinforcement Learningcited by 439
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2017Deep Reinforcement Learning: An Overviewcited by 327
  9. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  10. 2016Fairness in Learning: Classic and Contextual Banditscited by 167
  11. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 150
  12. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
27 more
  1. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  2. 2022Robust Reinforcement Learning using Offline Datacited by 95
  3. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  4. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 82
  5. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  6. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  7. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  8. 2021Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approachcited by 62
  9. 2021FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Financecited by 57
  10. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  11. 2022Personalized Decision Making -- A Conceptual Introductioncited by 29
  12. 2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
  13. 2021Fast Rates for the Regret of Offline Reinforcement Learningcited by 26
  14. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  15. 2022The Health Gym: Synthetic Health-Related Datasets for the Development of Reinforcement Learning Algorithmscited by 19
  16. 2020Robust Reinforcement Learning with Wasserstein Constraintcited by 18
  17. 2022What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatmentcited by 14
  18. 2020Fair Policy Targetingcited by 12
  19. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  20. 2020Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policycited by 7
  21. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  22. 2022On Learning and Testing of Counterfactual Fairness through Data Preprocessingcited by 6
  23. 2025Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessingcited by 5
  24. 2023Principal Stratification with Continuous Post-Treatment Variables: Nonparametric Identification and Semiparametric Estimationcited by 3
  25. 2024Semiparametric Localized Principal Stratification Analysis with Continuous Stratacited by 3
  26. 2025Fairness-aware Contextual Dynamic Pricing with Strategic Buyerscited by 2
  27. 2025The Promises of Multiple Experiments: Identifying Joint Distribution of Potential Outcomescited by 1

Built on by

Not cited yet.