mlsanityData to 7 Oct 2026

On the Role of Discount Factor in Offline Reinforcement Learning

arXiv 2206.03383 ↗First posted 7 Jun 2022cs.LGcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) enables effective learning from previously collected data without exploration, which shows great promise in real-world applications when exploration is expensive or even infeasible. The discount factor, γ, plays a vital role in improving online RL sample efficiency and estimation accuracy, but the role of the discount factor in offline RL is not well explored. This paper examines two distinct effects of γ in offline RL with theoretical analysis, namely the regularization effect and the pessimism effect. On the one hand, γ is a regulator to trade-off optimality with sample efficiency upon existing offline techniques. On the other hand, lower guidance γ can also be seen as a way of pessimism where we optimize the policy's performance in the worst possible models. We empirically verify the above theoretical observation with tabular MDPs and standard D4RL tasks. The results show that the discount factor plays an essential role in the performance of offline RL algorithms, both under small data regimes upon existing offline methods and in large data regimes without other conservative methods.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 49 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

36 of 49 references.

Closest ancestors · 19

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Comparison and Unification of Three Regularization Methods in Batch Reinforcement Learning16 Sep
  4. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  5. 2021Offline RL Without Off-Policy Evaluation16 Jun
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
7 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 17

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  7. 2018Meta-Gradient Reinforcement Learningcited by 156
  8. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  9. 2020Model-Based Offline Planningcited by 126
  10. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  11. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  12. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
5 more
  1. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
  2. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 44
  3. 2019Separating value functions across time-scalescited by 15
  4. 2020A Deeper Look at Discounting Mismatch in Actor-Critic Algorithmscited by 10
  5. 2019Gamma-Nets: Generalizing Value Estimation over Timescalecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 3; 2025: 2; 2026: 3.

16 papers.

Sort:

By area:

  1. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  3. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  4. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  5. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  6. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  7. 2023Improving Offline RL by Blending Heuristicscited by 10
  8. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  9. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
  10. 2023On the Value of Myopic Behavior in Policy Reusecited by 1
  11. 2024Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learningcited by 1
  12. 2026AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learningcited by 1
4 more
  1. 2023On the Effective Horizon of Inverse Reinforcement Learningcited by 0
  2. 2024Offline Imitation Learning by Controlling the Effective Planning Horizoncited by 0
  3. 2026Green or Fast? Learning to Balance Cold Starts and Idle Carbon in Serverless Computingcited by 0
  4. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  4. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  5. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
  6. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
  7. 2022Leveraging Offline Data in Online Reinforcement Learning
  8. 2020Hyperparameter Selection for Offline Reinforcement Learning