# A Risk-Averse Framework for Non-Stationary Stochastic Multi-Armed Bandits

arXiv: https://arxiv.org/abs/2310.19821 · HTML: https://mlsanity.com/paper/2310.19821/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 8 of 8)

- [Exploration vs Exploitation vs Safety: Risk-averse Multi-Armed Bandits](https://mlsanity.com/paper/1401.1123/) (2014)
- [Risk-Averse Multi-Armed Bandit Problems under Mean-Variance Measure](https://mlsanity.com/paper/1604.05257/) (2016)
- [Efficient Change-Point Detection for Tackling Piecewise-Stationary Bandits](https://mlsanity.com/paper/1902.01575/) (2019)
- [Generalized Risk-Aversion in Stochastic Multi-Armed Bandits](https://mlsanity.com/paper/1405.0833/) (2014)
- [Max K-armed bandit: On the ExtremeHunter algorithm and beyond](https://mlsanity.com/paper/1707.08820/) (2017)
- [A Survey of Risk-Aware Multi-Armed Bandits](https://mlsanity.com/paper/2205.05843/) (2022)
- [X-Armed Bandits: Optimizing Quantiles, CVaR and Other Risks](https://mlsanity.com/paper/1904.08205/) (2019)
- [Concentration bounds for CVaR estimation: The cases of light-tailed and heavy-tailed distributions](https://mlsanity.com/paper/1901.00997/) (2019)

## Abstract (arXiv)

In a typical stochastic multi-armed bandit problem, the objective is often to maximize the expected sum of rewards over some time horizon T. While the choice of a strategy that accomplishes that is optimal with no additional information, it is no longer the case when provided additional environment-specific knowledge. In particular, in areas of high volatility like healthcare or finance, a naive reward maximization approach often does not accurately capture the complexity of the learning problem and results in unreliable solutions. To tackle problems of this nature, we propose a framework of adaptive risk-aware strategies that operate in non-stationary environments. Our framework incorporates various risk measures prevalent in the literature to map multiple families of multi-armed bandit algorithms into a risk-sensitive setting. In addition, we equip the resulting algorithms with the Restarted Bayesian Online Change-Point Detection (R-BOCPD) algorithm and impose a (tunable) forced exploration strategy to detect local (per-arm) switches. We provide finite-time theoretical guarantees and an asymptotic regret bound of order tilde O(√KT T) up to time horizon T with KT the total number of change-points. In practice, our framework compares favorably to the state-of-the-art in both synthetic and real-world environments and manages to perform efficiently with respect to both risk-sensitivity and non-stationarity.