# Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach

arXiv: https://arxiv.org/abs/1902.02893 · HTML: https://mlsanity.com/paper/1902.02893/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2019.

## Built on by (most cited first, 30 of 32)

- [MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE](https://mlsanity.com/paper/2507.21802/) (2025)
- [On the Expressivity of Markov Reward](https://mlsanity.com/paper/2111.00876/) (2021)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [Settling the Reward Hypothesis](https://mlsanity.com/paper/2212.10420/) (2022)
- [Behavior Alignment via Reward Function Optimization](https://mlsanity.com/paper/2310.19007/) (2023)
- [Reinforcement Learning-powered Semantic Communication via Semantic Similarity](https://mlsanity.com/paper/2108.12121/) (2021)
- [Three Dogmas of Reinforcement Learning](https://mlsanity.com/paper/2407.10583/) (2024)
- [ProActive: Self-Attentive Temporal Point Process Flows for Activity Sequences](https://mlsanity.com/paper/2206.05291/) (2022)
- [LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning](https://mlsanity.com/paper/2209.10341/) (2022)
- [On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks](https://mlsanity.com/paper/2401.14811/) (2024)
- [Utility Theory for Sequential Decision Making](https://mlsanity.com/paper/2206.13637/) (2022)
- [Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards](https://mlsanity.com/paper/2310.00435/) (2023)
- [Emergent Risk Awareness in Rational Agents under Resource Constraints](https://mlsanity.com/paper/2505.23436/) (2025)
- [Online Reinforcement Learning with Uncertain Episode Lengths](https://mlsanity.com/paper/2302.03608/) (2023)
- [A Tale of Sampling and Estimation in Discounted Reinforcement Learning](https://mlsanity.com/paper/2304.05073/) (2023)
- [On the Expressivity of Multidimensional Markov Reward](https://mlsanity.com/paper/2307.12184/) (2023)
- [UGAE: A Novel Approach to Non-exponential Discounting](https://mlsanity.com/paper/2302.05740/) (2023)
- [The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning](https://mlsanity.com/paper/2306.11208/) (2023)
- [EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization](https://mlsanity.com/paper/2110.13373/) (2021)
- [Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPs](https://mlsanity.com/paper/2505.12049/) (2025)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning](https://mlsanity.com/paper/2105.02580/) (2021)
- [Tapestry of Time and Actions: Modeling Human Activity Sequences using Temporal Point Process Flows](https://mlsanity.com/paper/2307.10305/) (2023)
- [On shallow planning under partial observability](https://mlsanity.com/paper/2407.15820/) (2024)
- [Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting](https://mlsanity.com/paper/2602.04131/) (2026)
- [AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning](https://mlsanity.com/paper/2605.06149/) (2026)
- [Draining the Energy Commons: Self-Defeating Over-Appropriation as a Coordination Failure in Agentic LLM Collectives](https://mlsanity.com/paper/2607.22188/) (2026)
- [Factors of Influence of the Overestimation Bias of Q-Learning](https://mlsanity.com/paper/2210.05262/) (2022)
- [Markov Decision Processes with Time-Varying Geometric Discounting](https://mlsanity.com/paper/2307.10491/) (2023)
- [From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models](https://mlsanity.com/paper/2507.18229/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) agents have traditionally been tasked with maximizing the value function of a Markov decision process (MDP), either in continuous settings, with fixed discount factor γ < 1, or in episodic settings, with γ = 1. While this has proven effective for specific tasks with well-defined objectives (e.g., games), it has never been established that fixed discounting is suitable for general purpose use (e.g., as a model of human preferences). This paper characterizes rationality in sequential decision making using a set of seven axioms and arrives at a form of discounting that generalizes traditional fixed discounting. In particular, our framework admits a state-action dependent "discount" factor that is not constrained to be less than 1, so long as there is eventual long run discounting. Although this broadens the range of possible preference structures in continuous settings, we show that there exists a unique "optimizing MDP" with fixed γ < 1 whose optimal value function matches the true utility of the optimal policy, and we quantify the difference between value and utility for suboptimal policies. Our work can be seen as providing a normative justification for (a slight generalization of) Martha White's RL task formalism (2017) and other recent departures from the traditional RL, and is relevant to task specification in RL, inverse RL and preference-based RL.