# Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes

arXiv: https://arxiv.org/abs/2401.01841 · HTML: https://mlsanity.com/paper/2401.01841/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 9 of 9)

- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles](https://mlsanity.com/paper/2001.11231/) (2020)
- [Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments](https://mlsanity.com/paper/2209.15090/) (2022)
- [Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version](https://mlsanity.com/paper/1904.10090/) (2019)
- [Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processes](https://mlsanity.com/paper/1706.06544/) (2017)
- [A Review of Incident Prediction, Resource Allocation, and Dispatch Models for Emergency Management](https://mlsanity.com/paper/2006.04200/) (2020)
- [Dynamic Simplex: Balancing Safety and Performance in Autonomous Cyber Physical Systems](https://mlsanity.com/paper/2302.09750/) (2023)
- [Safety-Assured Speculative Planning with Adaptive Prediction](https://mlsanity.com/paper/2307.11876/) (2023)

## Built on by (most cited first, 4 of 4)

- [Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction](https://mlsanity.com/paper/2502.21186/) (2025)
- [Shrinking POMCP: A Framework for Real-Time UAV Search and Rescue](https://mlsanity.com/paper/2411.12967/) (2024)
- [SNAPE-PM: Building and Utilizing Dynamic Partner Models for Adaptive Explanation Generation](https://mlsanity.com/paper/2505.13053/) (2025)
- [Toward Template-Free Explainability for Monte Carlo Tree Search](https://mlsanity.com/paper/2605.16524/) (2026)

## Abstract (arXiv)

A fundamental (and largely open) challenge in sequential decision-making is dealing with non-stationary environments, where exogenous environmental conditions change over time. Such problems are traditionally modeled as non-stationary Markov decision processes (NSMDP). However, existing approaches for decision-making in NSMDPs have two major shortcomings: first, they assume that the updated environmental dynamics at the current time are known (although future dynamics can change); and second, planning is largely pessimistic, i.e., the agent acts ``safely'' to account for the non-stationary evolution of the environment. We argue that both these assumptions are invalid in practice -- updated environmental conditions are rarely known, and as the agent interacts with the environment, it can learn about the updated dynamics and avoid being pessimistic, at least in states whose dynamics it is confident about. We present a heuristic search algorithm called textitAdaptive Monte Carlo Tree Search (ADA-MCTS) that addresses these challenges. We show that the agent can learn the updated dynamics of the environment over time and then act as it learns, i.e., if the agent is in a region of the state space about which it has updated knowledge, it can avoid being pessimistic. To quantify ``updated knowledge,'' we disintegrate the aleatoric and epistemic uncertainty in the agent's updated belief and show how the agent can use these estimates for decision-making. We compare the proposed approach with the multiple state-of-the-art approaches in decision-making across multiple well-established open-source problems and empirically show that our approach is faster and highly adaptive without sacrificing safety.