mlsanityData to 7 Oct 2026

Model-Free Robust Average-Reward Reinforcement Learning

arXiv 2305.10504 ↗First posted 17 May 2023cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Robust Markov decision processes (MDPs) address the challenge of model uncertainty by optimizing the worst-case performance over an uncertainty set of MDPs. In this paper, we focus on the robust average-reward MDPs under the model-free setting. We first theoretically characterize the structure of solutions to the robust average-reward Bellman equation, which is essential for our later convergence analysis. We then design two model-free algorithms, robust relative value iteration (RVI) TD and robust RVI Q-learning, and theoretically prove their convergence to the optimal solution. We provide several widely used uncertainty sets as examples, including those defined by the contamination model, total variation, Chi-squared divergence, Kullback-Leibler (KL) divergence and Wasserstein distance.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 21 of its 55 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

21 of 55 references.

References · 21

  1. 2016OpenAI Gymcited by 1,684
  2. 2016Distributionally Robust Stochastic Optimization with Wasserstein Distancecited by 182
  3. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  4. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  5. 2022Robust Reinforcement Learning using Offline Datacited by 95
  6. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  7. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  8. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  9. 2017Reinforcement Learning under Model Mismatchcited by 52
  10. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  11. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
  12. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
9 more
  1. 2022Stochastic first-order methods for average-reward Markov decision processescited by 30
  2. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  3. 2022Learning Infinite-Horizon Average-Reward Markov Decision Processes with Constraintscited by 29
  4. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  5. 2018Robust Markov Decision Process: Beyond Rectangularitycited by 24
  6. 2023Robust Average-Reward Markov Decision Processescited by 22
  7. 2019Unbiased Multilevel Monte Carlo: Stochastic Optimization, Steady-state Simulation, Quantiles, and Other Applicationscited by 17
  8. 2022Unbiased Multilevel Monte Carlo methods for intractable distributions: MLMC meets MCMCcited by 5
  9. 2022On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  2. 2024Model-Free Robust Reinforcement Learning with Sample Complexity Analysiscited by 9
  3. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 6
  4. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learningcited by 6
  5. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
  6. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
  7. 2025Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gaincited by 3
  8. 2025From Abstraction to Reality: DARPA's Vision for Robust Sim-to-Real Autonomycited by 2
  9. 2025Planning and Learning in Average Risk-aware MDPscited by 2
  10. 2025Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysiscited by 2
  11. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  12. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learningcited by 2
8 more
  1. 2025Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPscited by 1
  2. 2025Provably Efficient Sample Complexity for Robust CMDPcited by 1
  3. 2026Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterizationcited by 1
  4. 2026Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learningcited by 1
  5. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processescited by 0
  6. 2025Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectivescited by 0
  7. 2026Robust Transfer Learning with Side Informationcited by 0
  8. 2026Stationary Robust Mean-Field Games under Model Mismatchescited by 0