# A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problems

arXiv: https://arxiv.org/abs/2110.14295 · HTML: https://mlsanity.com/paper/2110.14295/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 63 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/1703.01732/) (2017)
- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics](https://mlsanity.com/paper/1604.04057/) (2016)
- [Variance Adjusted Actor Critic Algorithms](https://mlsanity.com/paper/1310.3697/) (2013)
- [Mean-Variance Policy for Discrete-time Cone Constrained Markets: The Consistency in Efficiency and Minimum-Variance Signed Supermartingale Measure](https://mlsanity.com/paper/1403.0718/) (2014)
- [Weak equilibria for time-inconsistent control: with applications to investment-withdrawal decisions](https://mlsanity.com/paper/2105.06607/) (2021)
- [Nonlocal Fully Nonlinear Parabolic Differential Equations Arising in Time-Inconsistent Problems](https://mlsanity.com/paper/2110.04237/) (2021)

## Built on by (most cited first, 1 of 1)

- [Markov Decision Processes with Time-Varying Geometric Discounting](https://mlsanity.com/paper/2307.10491/) (2023)

## Abstract (arXiv)

In this paper, we establish a subgame perfect equilibrium reinforcement learning (SPERL) framework for time-inconsistent (TIC) problems. In the context of RL, TIC problems are known to face two main challenges: the non-existence of natural recursive relationships between value functions at different time points and the violation of Bellman's principle of optimality that raises questions on the applicability of standard policy iteration algorithms for unprovable policy improvement theorems. We adapt an extended dynamic programming theory and propose a new class of algorithms, called backward policy iteration (BPI), that solves SPERL and addresses both challenges. To demonstrate the practical usage of BPI as a training framework, we adapt standard RL simulation methods and derive two BPI-based training algorithms. We examine our derived training frameworks on a mean-variance portfolio selection problem and evaluate some performance metrics including convergence and model identifiability.