mlsanityData to 7 Oct 2026

On the Foundation of Distributionally Robust Reinforcement Learning

arXiv 2311.09018 ↗First posted 15 Nov 2023cs.LG cs.SY eess.SYcited by 18

Not on the map. Why?

Abstract (arXiv)

Motivated by the need for a robust policy in the face of environment shifts between training and deployment, we contribute to the theoretical foundation of distributionally robust reinforcement learning (DRRL). This is accomplished through a comprehensive modeling framework centered around robust Markov decision processes (RMDPs). This framework obliges the decision maker to choose an optimal policy under the worst-case distributional shift orchestrated by an adversary. By unifying and extending existing formulations, we rigorously construct RMDPs that embrace various modeling attributes for both the decision maker and the adversary. These attributes include the structure of information availability-covering history-dependent, Markov, and Markov time-homogeneous dynamics-as well as constraints on the shifts induced by the adversary, with a focus on SA- and S-rectangularity. Within this RMDP framework, we investigate conditions for the existence or absence of the dynamic programming principle (DPP). From an algorithmic standpoint, the existence of DPP holds significant implications, as the vast majority of existing data and computationally efficient DRRL algorithms are reliant on the DPP. To investigate its existence, we systematically analyze various combinations of controller and adversary attributes, presenting streamlined proofs based on a unified methodology. We then construct counterexamples for settings where a fully general DPP fails to hold and establish asymptotically optimal history-dependent policies for key scenarios where the DPP is absent.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 24 of its 68 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

24 of 68 references.

References · 24

  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  2. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  3. 2021Reinforcement learning based recommender systems: A surveycited by 187
  4. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 90
  5. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  6. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  7. 2017Virtual to Real Reinforcement Learning for Autonomous Drivingcited by 79
  8. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  9. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  10. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  11. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 35
  12. 2018Deep Reinforcement Learning for Sponsored Search Real-time Biddingcited by 34
12 more
  1. 2023Sample Complexity of Variance-reduced Distributionally Robust Q-learningcited by 28
  2. 2018Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approachcited by 27
  3. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  4. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  5. 2020Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Controlcited by 15
  6. 2017Learning-based Robust Optimization: Procedures and Statistical Guaranteescited by 9
  7. 2023Distributionally Robust Linear Quadratic Controlcited by 8
  8. 2020Recurrent Neural Networks for Stochastic Control in Real-Time Biddingcited by 6
  9. 2021Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approachcited by 6
  10. 2023Rectangularity and duality of distributionally robust Markov Decision Processescited by 6
  11. 2021Mathematical Foundations of Distributionally Robust Multistage Optimizationcited by 3
  12. 2017A Study of Distributionally Robust Multistage Stochastic Optimizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  2. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  3. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  4. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  5. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 12
  6. 2024Evaluating Model Performance Under Worst-case Subpopulationscited by 5
  7. 2025Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gaincited by 3
  8. 2024Do stable neural networks exist for classification problems? -- A new view on stability in AIcited by 2
  9. 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mappingcited by 2
  10. 2025Statistical and Algorithmic Foundations of Reinforcement Learningcited by 2
  11. 2025Near-Optimal Sample Complexities of Divergence-based S-rectangular Distributionally Robust Reinforcement Learningcited by 1
  12. 2025Mitigating Goal Misgeneralization via Minimax Regretcited by 1
6 more
  1. 2025Central Limit Theorems for Transition Probabilities of Controlled Markov Chainscited by 1
  2. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 1
  3. 2025Duality and Policy Evaluation in Distributionally Robust Bayesian Diffusion Controlcited by 0
  4. 2026Non-Rectangular Average-Reward Robust MDPs: Optimal Policies and Their Transient Valuescited by 0
  5. 2026Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approachcited by 0
  6. 2026Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RLcited by 0