# Finite Horizon Q-learning: Stability, Convergence, Simulations and an application on Smart Grids

arXiv: https://arxiv.org/abs/2110.15093 · HTML: https://mlsanity.com/paper/2110.15093/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 17 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 3 of 3)

- [Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning](https://mlsanity.com/paper/1909.03906/) (2019)
- [Generalized Speedy Q-learning](https://mlsanity.com/paper/1911.00397/) (2019)
- [A unified decision making framework for supply and demand management in microgrid networks](https://mlsanity.com/paper/1711.05078/) (2017)

## Built on by (most cited first, 3 of 3)

- [Tensor Low-rank Approximation of Finite-horizon Value Functions](https://mlsanity.com/paper/2405.17628/) (2024)
- [Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality](https://mlsanity.com/paper/2303.12785/) (2023)
- [On First-Order Meta-Reinforcement Learning with Moreau Envelopes](https://mlsanity.com/paper/2305.12216/) (2023)

## Abstract (arXiv)

Q-learning is a popular reinforcement learning algorithm. This algorithm has however been studied and analysed mainly in the infinite horizon setting. There are several important applications which can be modeled in the framework of finite horizon Markov decision processes. We develop a version of Q-learning algorithm for finite horizon Markov decision processes (MDP) and provide a full proof of its stability and convergence. Our analysis of stability and convergence of finite horizon Q-learning is based entirely on the ordinary differential equations (O.D.E) method. We also demonstrate the performance of our algorithm on a setting of random MDP as well as on an application on smart grids.