mlsanityData to 14 Oct 2026

Budgeting Counterfactual for Offline RL

arXiv 2307.06328 ↗First posted 12 Jul 2023cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The main challenge of offline reinforcement learning, where data is limited, arises from a sequence of counterfactual reasoning dilemmas within the realm of potential actions: What if we were to choose a different course of action? These circumstances frequently give rise to extrapolation errors, which tend to accumulate exponentially with the problem horizon. Hence, it becomes crucial to acknowledge that not all decision steps are equally important to the final outcome, and to budget the number of counterfactual decisions a policy make in order to control the extrapolation. Contrary to existing approaches that use regularization on either the policy or value function, we propose an approach to explicitly bound the amount of out-of-distribution actions during training. Specifically, our method utilizes dynamic programming to decide where to extrapolate and where not to, with an upper bound on the decisions different from behavior policy. It balances between the potential for improvement from taking out-of-distribution actions and the risk of making errors due to extrapolation. Theoretically, we justify our method by the constrained optimality of the fixed point solution to our Q updating rules. Empirically, we show that the overall performance of our method is better than the state-of-the-art offline RL methods on tasks in the widely-used D4RL benchmarks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 60 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

44 of 60 references.

Closest ancestors · 27

Same direction, newest first

  1. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms1 Jun
  2. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  3. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  4. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  5. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learning21 Nov
  6. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data8 Nov
  7. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  8. 2022Offline Policy Optimization with Eligible Actions1 Jul
  9. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  10. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  11. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
15 more
  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  2. 2021Offline RL Without Off-Policy Evaluation16 Jun
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  6. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  8. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  9. 2020Critic Regularized Regression26 Jun
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOPO: Model-based Offline Policy Optimization27 May
  12. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  13. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  14. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  15. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  10. 2019Batch Policy Learning under Constraintscited by 244
  11. 2022Online Decision Transformercited by 196
  12. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
5 more
  1. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  2. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 75
  3. 2020Off-policy Bandits with Deficient Supportcited by 61
  4. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  5. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  2. 2024AlphaRouter: Quantum Circuit Routing with Reinforcement Learning and Tree Searchcited by 8
  3. 2025Learning Counterfactual Outcomes Under Rank Preservationcited by 0
  4. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  2. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  3. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  4. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  5. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  6. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?
  7. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
  8. 2023Offline Reinforcement Learning with Additional Covering Distributions