# Contextual Restless Multi-Armed Bandits with Application to Demand Response Decision-Making

arXiv: https://arxiv.org/abs/2403.15640 · HTML: https://mlsanity.com/paper/2403.15640/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 30 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges](https://mlsanity.com/paper/1507.08025/) (2015)
- [Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges](https://mlsanity.com/paper/2102.01168/) (2021)
- [A Survey on Practical Applications of Multi-Armed and Contextual Bandits](https://mlsanity.com/paper/1904.10040/) (2019)
- [A Survey on Contextual Multi-armed Bandits](https://mlsanity.com/paper/1508.03326/) (2015)
- [Deadline Scheduling as Restless Bandits](https://mlsanity.com/paper/1610.00399/) (2016)
- [Online Residential Demand Response via Contextual Multi-Armed Bandits](https://mlsanity.com/paper/2003.03627/) (2020)
- [Exponential Convergence Rate for the Asymptotic Optimality of Whittle Index Policy](https://mlsanity.com/paper/2012.09064/) (2020)
- [Online Learning and Distributed Control for Residential Demand Response](https://mlsanity.com/paper/2010.05153/) (2020)
- [A Reliability-aware Multi-armed Bandit Approach to Learn and Select Users in Demand Response](https://mlsanity.com/paper/2003.09505/) (2020)

## Built on by (most cited first, 7 of 7)

- [Optimizing Vital Sign Monitoring in Resource-Constrained Maternal Care: An RL-Based Restless Bandit Approach](https://mlsanity.com/paper/2410.08377/) (2024)
- [Behavioral Generative Agents for Energy Operations](https://mlsanity.com/paper/2506.12664/) (2025)
- [Contextual Budget Bandit for Food Rescue Volunteer Engagement](https://mlsanity.com/paper/2509.10777/) (2025)
- [Neural Index Policies for Restless Multi-Action Bandits with Heterogeneous Budgets](https://mlsanity.com/paper/2510.22069/) (2025)
- [A Practical Algorithm for Feature-Rich, Non-Stationary Bandit Problems](https://mlsanity.com/paper/2603.16755/) (2026)
- [Set Membership Estimation for Stochastic Linear Contextual Bandits](https://mlsanity.com/paper/2606.20022/) (2026)
- [Lag: A Mean-Field Optimal Policy for Online Assignment with Reusable Resources](https://mlsanity.com/paper/2610.19038/) (2026)

## Abstract (arXiv)

This paper introduces a novel multi-armed bandits framework, termed Contextual Restless Bandits (CRB), for complex online decision-making. This CRB framework incorporates the core features of contextual bandits and restless bandits, so that it can model both the internal state transitions of each arm and the influence of external global environmental contexts. Using the dual decomposition method, we develop a scalable index policy algorithm for solving the CRB problem, and theoretically analyze the asymptotical optimality of this algorithm. In the case when the arm models are unknown, we further propose a model-based online learning algorithm based on the index policy to learn the arm models and make decisions simultaneously. Furthermore, we apply the proposed CRB framework and the index policy algorithm specifically to the demand response decision-making problem in smart grids. The numerical simulations demonstrate the performance and efficiency of our proposed CRB approaches.