# Solving Markov Decision Processes with Future Information via MPC

arXiv: https://arxiv.org/abs/2606.24991 · HTML: https://mlsanity.com/paper/2606.24991/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Cautious Model Predictive Control using Gaussian Process Regression](https://mlsanity.com/paper/1705.10702/) (2017)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Bayesian model predictive control: Efficient model exploration and regret bounds using posterior sampling](https://mlsanity.com/paper/2005.11744/) (2020)
- [All AI Models are Wrong, but Some are Optimal](https://mlsanity.com/paper/2501.06086/) (2025)
- [Data-Driven Predictive Control and MPC: Do we achieve optimality?](https://mlsanity.com/paper/2405.17892/) (2024)
- [On Incorporating Forecasts into Linear State Space Model Markov Decision Processes](https://mlsanity.com/paper/2103.07533/) (2021)
- [Economic Model Predictive Control as a Solution to Markov Decision Processes](https://mlsanity.com/paper/2407.16500/) (2024)

## Abstract (arXiv)

Model Predictive Control (MPC) is widely used in industrial and robotic systems for enforcing constraints and embedding domain knowledge through finite-horizon optimization-based planning. However, despite these strengths, an MPC scheme typically does not yield optimal policies for sequential decision-making problems formulated as Markov Decision Processes (MDPs). Recent combinations of MPC with Reinforcement Learning (RL) alleviate this issue by treating MPC as a parameterized model of the optimal policy of an MDP and adjusting its parameters using data. While these approaches typically consider classical MDPs, many real-world problems include future information--such as forecasts, prices, or reference trajectories--at decision time, which must be included in the MDP state for optimal decision-making. Current MPC-RL approaches do not directly account for this augmented-state structure, raising the question of how to incorporate future information into MPC to obtain an optimal policy. This work establishes the structural requirements under which a parameterized MPC can exactly represent the optimal value functions and policy of an MDP with future information. We further demonstrate that such a parameterized MPC can serve as a structured function approximator, with its parameters learned using RL. The approach is illustrated on a point-mass racing task with future reference information.