# When to Commit and When to Defer: Maturing Markov Decision Processes under Refining Information and Expiring Opportunities

arXiv: https://arxiv.org/abs/2606.18820 · HTML: https://mlsanity.com/paper/2606.18820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Learning Combinatorial Optimization Algorithms over Graphs](https://mlsanity.com/paper/1704.01665/) (2017)
- [Reinforcement Learning for Solving the Vehicle Routing Problem](https://mlsanity.com/paper/1802.04240/) (2018)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Parallelizing the dual revised simplex method](https://mlsanity.com/paper/1503.01889/) (2015)
- [Deep Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1511.04143/) (2015)
- [Reinforcement Learning with Parameterized Actions](https://mlsanity.com/paper/1509.01644/) (2015)
- [Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing](https://mlsanity.com/paper/2010.12001/) (2020)
- [DeepStock: Reinforcement Learning with Policy Regularizations for Inventory Management](https://mlsanity.com/paper/2603.19621/) (2026)
- [Maximum Weight Online Matching with Deadlines](https://mlsanity.com/paper/1808.03526/) (2018)
- [Stochastic Optimization with Parametric Cost Function Approximations](https://mlsanity.com/paper/1703.04644/) (2017)
- [Learning to Cover: Online Learning and Optimization with Irreversible Decisions](https://mlsanity.com/paper/2406.14777/) (2024)

## Abstract (arXiv)

Sequential decisions often become better informed as action opportunities expire. We introduce Maturing Markov Decision Processes (MMDPs), a structured subclass of augmented finite-horizon Markov decision processes, for organizing when to commit and when to defer. The Expiring-Actions-First Principle provides a sufficient condition for deferral based on information gain and delay loss. This structure motivates stage-local policies, action abstraction, and policy-guided planning. Replenishment experiments show improved learning and test performance under the MMDP interface and, in the primary setting, a larger test-performance gain from forecast refinement. In synthetic cash management, MMDP improves PPO test returns across both network sizes; five-account equal-size mask controls show that action count alone does not explain its learned gain, and the advantage persists with action abstraction and policy-guided search. A production-scale cash-management case study adds complementary evidence. Together, these results support commitment timing as a useful inductive bias for learning and planning.