# Risk-Aware Decision Making in Restless Bandits: Theory and Algorithms for Planning and Learning

arXiv: https://arxiv.org/abs/2410.23029 · HTML: https://mlsanity.com/paper/2410.23029/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 68 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Learning to Optimize Via Posterior Sampling](https://mlsanity.com/paper/1301.2609/) (2013)
- [(More) Efficient Reinforcement Learning via Posterior Sampling](https://mlsanity.com/paper/1306.0940/) (2013)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Learning Unknown Markov Decision Processes: A Thompson Sampling Approach](https://mlsanity.com/paper/1709.04570/) (2017)
- [Robust Risk-Aware Reinforcement Learning](https://mlsanity.com/paper/2108.10403/) (2021)
- [Thompson Sampling in Non-Episodic Restless Bandits](https://mlsanity.com/paper/1910.05654/) (2019)
- [Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents](https://mlsanity.com/paper/2301.12601/) (2023)
- [Opportunistic Scheduling as Restless Bandits](https://mlsanity.com/paper/1706.09778/) (2017)
- [On learning Whittle index policy for restless bandits with scalable regret](https://mlsanity.com/paper/2202.03463/) (2022)
- [Scalable Operator Allocation for Multi-Robot Assistance: A Restless Bandit Approach](https://mlsanity.com/paper/2111.06437/) (2021)
- [Dynamic priority allocation via restless bandit marginal productivity indices](https://mlsanity.com/paper/2304.06115/) (2023)
- [Reinforcement and Deep Reinforcement Learning-based Solutions for Machine Maintenance Planning, Scheduling Policies, and Optimization](https://mlsanity.com/paper/2307.03860/) (2023)
- [Whittle Index Policy for Crawling Ephemeral Content](https://mlsanity.com/paper/1503.08558/) (2015)
- [Risk-Sensitive Learning and Pricing for Demand Response](https://mlsanity.com/paper/1611.07098/) (2016)
- [A (2/3)n3 fast-pivoting algorithm for the Gittins index and optimal stopping of a Markov chain](https://mlsanity.com/paper/2304.01840/) (2023)

## Abstract (arXiv)

In restless bandits, a central agent is tasked with optimally distributing limited resources across several bandits (arms), with each arm being a Markov decision process. In this work, we generalize the traditional restless bandits problem with a risk-neutral objective by incorporating risk-awareness, which is particularly important in various real-world applications especially when the decision maker seeks to mitigate downside risks. We establish indexability conditions for the case of a risk-aware objective and provide a solution based on Whittle index for the first time for the planning problem with finite-horizon non-stationary and for infinite-horizon stationary Markov decision processes. In addition, we address the learning problem when the true transition probabilities are unknown by proposing a Thompson sampling approach and show that it achieves bounded regret that scales sublinearly with the number of episodes and quadratically with the number of arms. The efficacy of our method in reducing risk exposure in restless bandits is illustrated through a set of numerical experiments in the contexts of machine replacement and patient scheduling applications under both planning and learning setups.