# Markov Decision Processes with Time-Varying Geometric Discounting

arXiv: https://arxiv.org/abs/2307.10491 · HTML: https://mlsanity.com/paper/2307.10491/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 47 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 7 of 7)

- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [Unifying task specification in reinforcement learning](https://mlsanity.com/paper/1609.01995/) (2016)
- [Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach](https://mlsanity.com/paper/1902.02893/) (2019)
- [Separating value functions across time-scales](https://mlsanity.com/paper/1902.01883/) (2019)
- [On the Complexity of Value Iteration](https://mlsanity.com/paper/1807.04920/) (2018)
- [A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problems](https://mlsanity.com/paper/2110.14295/) (2021)

## Abstract (arXiv)

Canonical models of Markov decision processes (MDPs) usually consider geometric discounting based on a constant discount factor. While this standard modeling approach has led to many elegant results, some recent studies indicate the necessity of modeling time-varying discounting in certain applications. This paper studies a model of infinite-horizon MDPs with time-varying discount factors. We take a game-theoretic perspective -- whereby each time step is treated as an independent decision maker with their own (fixed) discount factor -- and we study the subgame perfect equilibrium (SPE) of the resulting game as well as the related algorithmic problems. We present a constructive proof of the existence of an SPE and demonstrate the EXPTIME-hardness of computing an SPE. We also turn to the approximate notion of ε-SPE and show that an ε-SPE exists under milder assumptions. An algorithm is presented to compute an ε-SPE, of which an upper bound of the time complexity, as a function of the convergence property of the time-varying discount factor, is provided.