# Predictive Control and Regret Analysis of Non-Stationary MDP with Look-ahead Information

arXiv: https://arxiv.org/abs/2409.08434 · HTML: https://mlsanity.com/paper/2409.08434/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 19 of 19)

- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approach](https://mlsanity.com/paper/2102.05406/) (2021)
- [Online Optimal Control with Linear Dynamics and Predictions: Algorithms and Regret Analysis](https://mlsanity.com/paper/1906.11378/) (2019)
- [Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems](https://mlsanity.com/paper/2103.04490/) (2021)
- [Logarithmic Regret for Online Control](https://mlsanity.com/paper/1909.05062/) (2019)
- [Optimizing for the Future in Non-Stationary MDPs](https://mlsanity.com/paper/2005.08158/) (2020)
- [The Power of Predictions in Online Control](https://mlsanity.com/paper/2006.07569/) (2020)
- [A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitions](https://mlsanity.com/paper/1805.10066/) (2018)
- [Perturbation-based Regret Analysis of Predictive Control in Linear Time Varying Systems](https://mlsanity.com/paper/2106.10497/) (2021)
- [Nearly Minimax Optimal Regret for Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation](https://mlsanity.com/paper/2102.07301/) (2021)
- [Dynamic Regret of Online Markov Decision Processes](https://mlsanity.com/paper/2208.12483/) (2022)
- [Reinforcement Learning with Lookahead Information](https://mlsanity.com/paper/2406.02258/) (2024)
- [Efficient Reinforcement Learning for Routing Jobs in Heterogeneous Queueing Systems](https://mlsanity.com/paper/2402.01147/) (2024)
- [Non-stationary Reinforcement Learning under General Function Approximation](https://mlsanity.com/paper/2306.00861/) (2023)
- [Smoothed Least-Laxity-First Algorithm for EV Charging](https://mlsanity.com/paper/2102.08610/) (2021)
- [Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel](https://mlsanity.com/paper/2210.04098/) (2022)
- [Restarted Bayesian Online Change-point Detection for Non-Stationary Markov Decision Processes](https://mlsanity.com/paper/2304.00232/) (2023)
- [Pausing Policy Learning in Non-stationary Reinforcement Learning](https://mlsanity.com/paper/2405.16053/) (2024)
- [Optimistic Regret Bounds for Online Learning in Adversarial Markov Decision Processes](https://mlsanity.com/paper/2405.02188/) (2024)

## Built on by (most cited first, 2 of 2)

- [Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach](https://mlsanity.com/paper/2510.18687/) (2025)
- [Reinforcement Learning with Multi-Step Lookahead Information Via Adaptive Batching](https://mlsanity.com/paper/2601.10418/) (2026)

## Abstract (arXiv)

Policy design in non-stationary Markov Decision Processes (MDPs) is inherently challenging due to the complexities introduced by time-varying system transition and reward, which make it difficult for learners to determine the optimal actions for maximizing cumulative future rewards. Fortunately, in many practical applications, such as energy systems, look-ahead predictions are available, including forecasts for renewable energy generation and demand. In this paper, we leverage these look-ahead predictions and propose an algorithm designed to achieve low regret in non-stationary MDPs by incorporating such predictions. Our theoretical analysis demonstrates that, under certain assumptions, the regret decreases exponentially as the look-ahead window expands. When the system prediction is subject to error, the regret does not explode even if the prediction error grows sub-exponentially as a function of the prediction horizon. We validate our approach through simulations, confirming the efficacy of our algorithm in non-stationary environments.