# Continuous-in-time Limit for Bayesian Bandits

arXiv: https://arxiv.org/abs/2210.07513 · HTML: https://mlsanity.com/paper/2210.07513/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Stein Variational Gradient Descent as Gradient Flow](https://mlsanity.com/paper/1704.07520/) (2017)
- [Stochastic modified equations and adaptive stochastic gradient algorithms](https://mlsanity.com/paper/1511.06251/) (2015)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [A Dynamical Central Limit Theorem for Shallow Neural Networks](https://mlsanity.com/paper/2008.09623/) (2020)
- [Diffusion Approximations for Thompson Sampling in the Small Gap Regime](https://mlsanity.com/paper/2105.09232/) (2021)
- [Regret Analysis of the Finite-Horizon Gittins Index Strategy for Multi-Armed Bandits](https://mlsanity.com/paper/1511.06014/) (2015)
- [A Note on Optimization Formulations of Markov Decision Processes](https://mlsanity.com/paper/2012.09417/) (2020)
- [Diffusion Approximations for a Class of Sequential Testing Problems](https://mlsanity.com/paper/2102.07030/) (2021)
- [A PDE-Based Analysis of the Symmetric Two-Armed Bernoulli Bandit](https://mlsanity.com/paper/2202.05767/) (2022)
- [A New Approach to Drifting Games, Based on Asymptotically Optimal Potentials](https://mlsanity.com/paper/2207.11405/) (2022)

## Built on by (most cited first, 2 of 2)

- [A note on continuous-time online learning](https://mlsanity.com/paper/2405.10399/) (2024)
- [Trading Vector Data in Vector Databases](https://mlsanity.com/paper/2511.07139/) (2025)

## Abstract (arXiv)

This paper revisits the bandit problem in the Bayesian setting. The Bayesian approach formulates the bandit problem as an optimization problem, and the goal is to find the optimal policy which minimizes the Bayesian regret. One of the main challenges facing the Bayesian approach is that computation of the optimal policy is often intractable, especially when the length of the problem horizon or the number of arms is large. In this paper, we first show that under a suitable rescaling, the Bayesian bandit problem converges toward a continuous Hamilton-Jacobi-Bellman (HJB) equation. The optimal policy for the limiting HJB equation can be explicitly obtained for several common bandit problems, and we give numerical methods to solve the HJB equation when an explicit solution is not available. Based on these results, we propose an approximate Bayes-optimal policy for solving Bayesian bandit problems with large horizons. Our method has the added benefit that its computational cost does not increase as the horizon increases.