mlsanityData to 7 Oct 2026

Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version

arXiv 1904.10090 ↗First posted 22 Apr 2019cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

This work tackles the problem of robust zero-shot planning in non-stationary stochastic environments. We study Markov Decision Processes (MDPs) evolving over time and consider Model-Based Reinforcement Learning algorithms in this setting. We make two hypotheses: 1) the environment evolves continuously with a bounded evolution rate; 2) a current model is known at each decision epoch but not its evolution. Our contribution can be presented in four points. 1) we define a specific class of MDPs that we call Non-Stationary MDPs (NSMDPs). We introduce the notion of regular evolution by making an hypothesis of Lipschitz-Continuity on the transition and reward functions w.r.t. time; 2) we consider a planning agent using the current model of the environment but unaware of its future evolution. This leads us to consider a worst-case method where the environment is seen as an adversarial agent; 3) following this approach, we propose the Risk-Averse Tree-Search (RATS) algorithm, a zero-shot Model-Based method similar to Minimax search; 4) we illustrate the benefits brought by RATS empirically and compare its performance with reference Model-Based algorithms.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 5; 2022: 9; 2023: 7; 2024: 5; 2025: 3; 2026: 6.

36 papers.

Sort:

By area:

  1. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  2. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 40
  3. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  4. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  5. 2021Continual Learning In Environments With Polynomial Mixing Timescited by 13
  6. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  7. 2022Dynamic Dialogue Policy for Continual Reinforcement Learningcited by 12
  8. 2022Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learningcited by 9
  9. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  10. 2021Explicit Explore, Exploit, or Escape (E4): near-optimal safety-constrained reinforcement learning in polynomial timecited by 6
  11. 2022Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variablescited by 6
  12. 2021Lifetime policy reuse and the importance of task capacitycited by 4
24 more
  1. 2024Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processescited by 4
  2. 2024Decision Making in Non-Stationary Environments with Policy-Augmented Searchcited by 4
  3. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  4. 2023SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learningcited by 3
  5. 2023Reinforcement learning-based estimation for partial differential equationscited by 3
  6. 2021Coarse-Grained Smoothness for RL in Metric Spacescited by 2
  7. 2023Efficient Action Robust Reinforcement Learning with Probabilistic Policy Execution Uncertaintycited by 2
  8. 2024Time-Constrained Robust MDPscited by 2
  9. 2023Reinforcement Learning under External Influence: Guarantees, Algorithms, and Sample Complexitycited by 1
  10. 2024Tolerance of Reinforcement Learning Controllers against Deviations in Cyber Physical Systemscited by 1
  11. 2024Solving robust MDPs as a sequence of static RL problemscited by 1
  12. 2026Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Artcited by 1
  13. 2026Non-Stationary Online Resource Allocation: Learning from a Single Samplecited by 1
  14. 2022Non-Parametric Stochastic Policy Gradient with Strategic Retreat for Non-Stationary Environmentcited by 0
  15. 2022Set-based value operators for non-stationary Markovian environmentscited by 0
  16. 2022Prioritizing emergency evacuations under compounding levels of uncertaintycited by 0
  17. 2023Model-Free Learning and Optimal Policy Design in Multi-Agent MDPs Under Probabilistic Agent Dropoutcited by 0
  18. 2025Natural Policy Gradient for Average Reward Non-Stationary RLcited by 0
  19. 2025SNAPE-PM: Building and Utilizing Dynamic Partner Models for Adaptive Explanation Generationcited by 0
  20. 2025A Translation of Probabilistic Event Calculus into Markov Decision Processescited by 0
  21. 2026Safe Continual Reinforcement Learning in Non-stationary Environmentscited by 0
  22. 2026BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous controlcited by 0
  23. 2026Reinforcement Learning in Operational Research: A Technical Review and Practical Roadmapcited by 0
  24. 2026AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learningcited by 0