# Improvements and Generalizations of Stochastic Knapsack and Multi-Armed Bandit Approximation Algorithms: Full Version

arXiv: https://arxiv.org/abs/1306.1149 · HTML: https://mlsanity.com/paper/1306.1149/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2013.

## Built on by (most cited first, 1 of 1)

- [Learning to Explore with Lagrangians for Bandits under Unknown Linear Constraints](https://mlsanity.com/paper/2410.18844/) (2024)

## Abstract (arXiv)

We study the multi-armed bandit problem with arms which are Markov chains with rewards. In the finite-horizon setting, the celebrated Gittins indices do not apply, and the exact solution is intractable. We provide approximation algorithms for a more general model which includes Markov decision processes and non-unit transition times. When preemption is allowed, we provide a (1/2-eps)-approximation, along with an example showing this is tight. When preemption isn't allowed, we provide a 1/12-approximation, which improves to a 4/27-approximation when transition times are unity. Our model encompasses the Markovian Bandits model of Gupta et al, the Stochastic Knapsack model of Dean, Goemans, and Vondrak, and the Budgeted Learning model of Guha and Munagala, and our algorithms improve existing results in all three areas. In our analysis, we encounter and overcome to our knowledge a novel obstacle - an algorithm that provably exists via polyhedral arguments, but cannot be found in polynomial time.