mlsanityData to 14 Oct 2026

Time-Constrained Robust MDPs

arXiv 2406.08395 ↗First posted 12 Jun 2024cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Robust reinforcement learning is essential for deploying reinforcement learning algorithms in real-world scenarios where environmental uncertainty predominates. Traditional robust reinforcement learning often depends on rectangularity assumptions, where adverse probability measures of outcome states are assumed to be independent across different states and actions. This assumption, rarely fulfilled in practice, leads to overly conservative policies. To address this problem, we introduce a new time-constrained robust MDP (TC-RMDP) formulation that considers multifactorial, correlated, and time-dependent disturbances, thus more accurately reflecting real-world dynamics. This formulation goes beyond the conventional rectangularity paradigm, offering new perspectives and expanding the analytical framework for robust RL. We propose three distinct algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. Our results demonstrate that these algorithms yield an efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks. This study revisits the prevailing assumptions in robust RL and opens new avenues for developing more practical and realistic RL applications.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 33 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

19 of 33 references.

References · 19

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2019Solving Rubik's Cube with a Robot Handcited by 734
  4. 2017Robust Adversarial Reinforcement Learningcited by 451
  5. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  6. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  7. 2019Active Domain Randomizationcited by 170
  8. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  9. 2020Robust Reinforcement Learning using Adversarial Populationscited by 66
  10. 2019Wasserstein Robust Reinforcement Learningcited by 54
  11. 2019Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncited by 36
  12. 2018A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitionscited by 30
7 more
  1. 2018Robust Markov Decision Process: Beyond Rectangularitycited by 24
  2. 2013Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributionscited by 18
  3. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
  4. 2022Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecificationcited by 11
  5. 2019Learning and Planning for Time-Varying MDPs Using Maximum Likelihood Estimationcited by 10
  6. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 10
  7. 2019Non-Stationary Reinforcement Learning: The Blessing of (More) Optimismcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  2. 2026Reward-Preserving Attacks For Robust Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Solving robust MDPs as a sequence of static RL problems
  2. 2024Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model
  3. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
  4. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  5. 2026Robust Peak-cost Constrained Reinforcement Learning
  6. 2024RRLS : Robust Reinforcement Learning Suite
  7. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning
  8. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees