# Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gain

arXiv: https://arxiv.org/abs/2509.14203 · HTML: https://mlsanity.com/paper/2509.14203/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 18 of 18)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Online Reinforcement Learning in Stochastic Games](https://mlsanity.com/paper/1712.00579/) (2017)
- [Sharper Model-free Reinforcement Learning for Average-reward Markov Decision Processes](https://mlsanity.com/paper/2306.16394/) (2023)
- [Robust Average-Reward Markov Decision Processes](https://mlsanity.com/paper/2301.00858/) (2023)
- [Model-Free Robust Average-Reward Reinforcement Learning](https://mlsanity.com/paper/2305.10504/) (2023)
- [On the Foundation of Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2311.09018/) (2023)
- [First-Order Methods for Wasserstein Distributionally Robust MDP](https://mlsanity.com/paper/2009.06790/) (2020)
- [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023)
- [Beyond discounted returns: Robust Markov decision processes with average and Blackwell optimality](https://mlsanity.com/paper/2312.03618/) (2023)
- [An overview of systems-theoretic guarantees in data-driven model predictive control](https://mlsanity.com/paper/2406.04130/) (2024)
- [A Zero-Sum Stochastic Game with Compact Action Sets and no Asymptotic Value](https://mlsanity.com/paper/1301.4540/) (2013)
- [Zero-sum repeated games: Counterexamples to the existence of the asymptotic value and the conjecture operatornamemaxmin=operatornamelimvn](https://mlsanity.com/paper/1305.4778/) (2013)
- [Dynamic Programming-Compatible Uncertainty Sets in Robust Markov Decision Processes](https://mlsanity.com/paper/2411.08435/) (2024)
- [Playing against a stationary opponent](https://mlsanity.com/paper/2503.15346/) (2025)
- [Communicating Zero-Sum Product Stochastic Games](https://mlsanity.com/paper/1709.00898/) (2017)
- [A tutorial on Zero-sum Stochastic Games](https://mlsanity.com/paper/1905.06577/) (2019)
- [Thresholds for sensitive optimality and Blackwell optimality in stochastic games](https://mlsanity.com/paper/2506.18545/) (2025)

## Built on by (most cited first, 5 of 5)

- [Non-Rectangular Average-Reward Robust MDPs: Optimal Policies and Their Transient Values](https://mlsanity.com/paper/2603.00945/) (2026)
- [Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterization](https://mlsanity.com/paper/2602.11387/) (2026)
- [Stationary Robust Mean-Field Games under Model Mismatches](https://mlsanity.com/paper/2606.22579/) (2026)
- [Vector Bellman Theory for Multichain Robust Average-Reward Markov Decision Processes](https://mlsanity.com/paper/2609.28792/) (2026)
- [Distributionally Robust Average-Reward Reinforcement Learning: Finite-Sample Guarantees under Weak Communication](https://mlsanity.com/paper/2609.32727/) (2026)

## Abstract (arXiv)

Learning and optimal control under robust Markov decision processes (MDPs) have received increasing attention, yet most existing theory, algorithms, and applications focus on finite-horizon or discounted models. Long-run average-reward formulations, while natural in many operations research and management contexts, remain underexplored. This is primarily because the dynamic programming foundations are technically challenging and only partially understood, with several fundamental questions remaining open. This paper steps toward a general framework for average-reward robust MDPs by analyzing the constant-gain setting. We study the average-reward robust control problem with possible information asymmetries between the controller and an S-rectangular adversary. Our analysis centers on the constant-gain robust Bellman equation, examining both the existence of solutions and their relationship to the optimal average reward. Specifically, we identify when solutions to the robust Bellman equation characterize the optimal average reward and stationary policies, and we provide one-sided weak communication conditions ensuring solutions' existence. These findings expand the dynamic programming theory for average-reward robust MDPs and lay a foundation for robust dynamic decision making under long-run average criteria in operational environments.