# MDP modeling for multi-stage stochastic programs

arXiv: https://arxiv.org/abs/2509.22981 · HTML: https://mlsanity.com/paper/2509.22981/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 59 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 8 of 8)

- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [A General Framework for Updating Belief Distributions](https://mlsanity.com/paper/1306.6430/) (2013)
- [Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes](https://mlsanity.com/paper/2102.00135/) (2021)
- [JuMP 1.0: Recent improvements to a modeling language for mathematical optimization](https://mlsanity.com/paper/2206.03866/) (2022)
- [Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes](https://mlsanity.com/paper/1805.04419/) (2018)
- [Adaptive Two-stage Stochastic Programming with an Analysis on Capacity Expansion Planning Problem](https://mlsanity.com/paper/1906.03513/) (2019)
- [Risk Filtering and Risk-Averse Control of Markovian Systems Subject to Model Uncertainty](https://mlsanity.com/paper/2206.09235/) (2022)
- [Convergence analysis of sampling-based decomposition methods for risk-averse multistage stochastic convex programs](https://mlsanity.com/paper/1408.4439/) (2014)

## Built on by (most cited first, 1 of 1)

- [Deep Learning for Sequential Decision Making under Uncertainty: Foundations, Frameworks, and Frontiers](https://mlsanity.com/paper/2604.11507/) (2026)

## Abstract (arXiv)

We study a class of multi-stage stochastic programs, which incorporate modeling features from Markov decision processes (MDPs). This class includes structured MDPs with continuous action and state spaces. We extend policy graphs to include decision-dependent uncertainty for one-step transition probabilities as well as a limited form of statistical learning. We focus on the expressiveness of our modeling approach, illustrating ideas with a series of examples of increasing complexity. As a solution method, we develop new variants of stochastic dual dynamic programming, including approximations to handle non-convexities.