# A Direct Approach for Handling Contextual Bandits with Latent State Dynamics

arXiv: https://arxiv.org/abs/2604.08149 · HTML: https://mlsanity.com/paper/2604.08149/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 25 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 7 of 7)

- [Learning Contextual Bandits in a Non-stationary Environment](https://mlsanity.com/paper/1805.09365/) (2018)
- [The Elliptical Potential Lemma Revisited](https://mlsanity.com/paper/2010.10182/) (2020)
- [Tractable Inference for Complex Stochastic Processes](https://mlsanity.com/paper/1301.7362/) (2013)
- [Non-Stationary Latent Bandits](https://mlsanity.com/paper/2012.00386/) (2020)
- [Regime Switching Bandits](https://mlsanity.com/paper/2001.09390/) (2020)
- [Target Tracking for Contextual Bandits: Application to Demand Side Management](https://mlsanity.com/paper/1901.09532/) (2019)
- [A New Bandit Setting Balancing Information from State Evolution and Corrupted Context](https://mlsanity.com/paper/2011.07989/) (2020)

## Abstract (arXiv)

We consider a linear contextual bandit model where contexts and rewards are governed by a finite hidden Markov chain. We first revisit the simplified model by Nelson et al. (2022), in which rewards are linear functions of the posterior probabilities over the hidden states given the observed contexts (called beliefs), rather than functions of the hidden states themselves. This simplified model may be handled through a direct reduction to standard linear contextual bandits. We extend the theoretical analysis of this reduction to take into account the estimation of the parameters of the hidden Markov model [HMM] in the regret bound and to provide high-probability bounds not depending anymore on the reward functions and only depending on the model through the estimation of the HMM parameters. Second, and most importantly, we instead study the more natural and more complex model incorporating direct dependencies in the hidden states (on top of dependencies on the observed contexts, as is natural for contextual bandits). Under a classic HMM forgetting condition, the main algorithmic tool introduced to cope with the various statistical dependencies that the reward structure introduces is to only periodically update reward-model parameters.