# An empirical evaluation of active inference in multi-armed bandits

arXiv: https://arxiv.org/abs/2101.08699 · HTML: https://mlsanity.com/paper/2101.08699/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 112 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 16 of 16)

- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [An Information-Theoretic Analysis of Thompson Sampling](https://mlsanity.com/paper/1403.5341/) (2014)
- [Learning to Optimize via Information-Directed Sampling](https://mlsanity.com/paper/1403.5556/) (2014)
- [Active inference on discrete state-spaces: a synthesis](https://mlsanity.com/paper/2001.07203/) (2020)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [Sophisticated Inference](https://mlsanity.com/paper/2006.04120/) (2020)
- [Deep Active Inference as Variational Policy Gradients](https://mlsanity.com/paper/1907.03876/) (2019)
- [A Change-Detection based Framework for Piecewise-stationary Multi-Armed Bandit Problem](https://mlsanity.com/paper/1711.03539/) (2017)
- [Hedging the Drift: Learning to Optimize under Non-Stationarity](https://mlsanity.com/paper/1903.01461/) (2019)
- [Deep Active Inference](https://mlsanity.com/paper/1709.02341/) (2017)
- [Taming Non-stationary Bandits: A Bayesian Approach](https://mlsanity.com/paper/1707.09727/) (2017)
- [On Bayesian index policies for sequential resource allocation](https://mlsanity.com/paper/1601.01190/) (2016)
- [Efficient Change-Point Detection for Tackling Piecewise-Stationary Bandits](https://mlsanity.com/paper/1902.01575/) (2019)
- [Modeling Human Decision-making in Generalized Gaussian Multi-armed Bandits](https://mlsanity.com/paper/1307.6134/) (2013)
- [On Adaptive Estimation for Dynamic Bernoulli Bandits](https://mlsanity.com/paper/1712.03134/) (2017)

## Built on by (most cited first, 12 of 12)

- [On efficient computation in active inference](https://mlsanity.com/paper/2307.00504/) (2023)
- [Active Inference as a Model of Agency](https://mlsanity.com/paper/2401.12917/) (2024)
- [A taxonomy of surprise definitions](https://mlsanity.com/paper/2209.01034/) (2022)
- [Active Inference for Autonomous Decision-Making with Contextual Multi-Armed Bandits](https://mlsanity.com/paper/2209.09185/) (2022)
- [Chance-Constrained Active Inference](https://mlsanity.com/paper/2102.08792/) (2021)
- [Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents](https://mlsanity.com/paper/2203.10592/) (2022)
- [Observation-Augmented Contextual Multi-Armed Bandits for Robotic Search and Exploration](https://mlsanity.com/paper/2312.12583/) (2023)
- [Approximate information for efficient exploration-exploitation strategies](https://mlsanity.com/paper/2307.01563/) (2023)
- [Sophisticated Learning: A novel algorithm for active learning during model-based planning](https://mlsanity.com/paper/2308.08029/) (2023)
- [Realising Active Inference in Variational Message Passing: the Outcome-blind Certainty Seeker](https://mlsanity.com/paper/2104.11798/) (2021)
- [Maximum entropy exploration in contextual bandits with neural networks and energy based models](https://mlsanity.com/paper/2210.06302/) (2022)
- [Distributional Active Inference](https://mlsanity.com/paper/2601.20985/) (2026)

## Abstract (arXiv)

A key feature of sequential decision making under uncertainty is a need to balance between exploiting--choosing the best action according to the current knowledge, and exploring--obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration-exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.