# Blackwell Online Learning for Markov Decision Processes

arXiv: https://arxiv.org/abs/2012.14043 · HTML: https://mlsanity.com/paper/2012.14043/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Dec 2020.

## Built on by (most cited first, 12 of 12)

- [The Confluence of Networks, Games and Learning](https://mlsanity.com/paper/2105.08158/) (2021)
- [Self-Adaptive Driving in Nonstationary Environments through Conjectural Online Lookahead Adaptation](https://mlsanity.com/paper/2210.03209/) (2022)
- [Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2306.06766/) (2023)
- [Symbiotic Game and Foundation Models for Cyber Deception Operations in Strategic Cyber Warfare](https://mlsanity.com/paper/2403.10570/) (2024)
- [Multi-level Traffic-Responsive Tilt Camera Surveillance through Predictive Correlated Online Learning](https://mlsanity.com/paper/2408.02208/) (2024)
- [Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis](https://mlsanity.com/paper/2208.00081/) (2022)
- [Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.04579/) (2023)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [Digital Twin-based Driver Risk-Aware Intelligent Mobility Analytics for Urban Transportation Management](https://mlsanity.com/paper/2407.15025/) (2024)
- [Commitment with Signaling under Double-sided Information Asymmetry](https://mlsanity.com/paper/2212.11446/) (2022)
- [Neurosymbolic Meta-Reinforcement Lookahead Learning Achieves Safe Self-Driving in Non-Stationary Environments](https://mlsanity.com/paper/2309.02328/) (2023)
- [Time-Dependent Blackwell Approachability and Application to Absorbing Games](https://mlsanity.com/paper/2303.04956/) (2023)

## Abstract (arXiv)

This work provides a novel interpretation of Markov Decision Processes (MDP) from the online optimization viewpoint. In such an online optimization context, the policy of the MDP is viewed as the decision variable while the corresponding value function is treated as payoff feedback from the environment. Based on this interpretation, we construct a Blackwell game induced by MDP, which bridges the gap among regret minimization, Blackwell approachability theory, and learning theory for MDP. Specifically, from the approachability theory, we propose 1) Blackwell value iteration for offline planning and 2) Blackwell Q-learning for online learning in MDP, both of which are shown to converge to the optimal solution. Our theoretical guarantees are corroborated by numerical experiments.