# Deep Bayesian Bandits: Exploring in Online Personalized Recommendations

arXiv: https://arxiv.org/abs/2008.00727 · HTML: https://mlsanity.com/paper/2008.00727/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2020.

## Built on by (most cited first, 23 of 23)

- [Efficient Online Bayesian Inference for Neural Bandits](https://mlsanity.com/paper/2112.00195/) (2021)
- [Scalable Neural Contextual Bandit for Recommender Systems](https://mlsanity.com/paper/2306.14834/) (2023)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Online Learning for Recommendations at Grubhub](https://mlsanity.com/paper/2107.07106/) (2021)
- [Fast online inference for nonlinear contextual bandit based on Generative Adversarial Network](https://mlsanity.com/paper/2202.08867/) (2022)
- [Dynamic Slate Recommendation with Gated Recurrent Units and Thompson Sampling](https://mlsanity.com/paper/2104.15046/) (2021)
- [Adversarial Gradient Driven Exploration for Deep Click-Through Rate Prediction](https://mlsanity.com/paper/2112.11136/) (2021)
- [Deep Upper Confidence Bound Algorithm for Contextual Bandit Ranking of Information Selection](https://mlsanity.com/paper/2110.04127/) (2021)
- [Towards Efficient and Optimal Covariance-Adaptive Algorithms for Combinatorial Semi-Bandits](https://mlsanity.com/paper/2402.15171/) (2024)
- [Optimal Regret Is Achievable with Bounded Approximate Inference Error: An Enhanced Bayesian Upper Confidence Bound Framework](https://mlsanity.com/paper/2201.12955/) (2022)
- [Two-Stage Neural Contextual Bandits for Personalised News Recommendation](https://mlsanity.com/paper/2206.14648/) (2022)
- [Top-K Ranking Deep Contextual Bandits for Information Selection Systems](https://mlsanity.com/paper/2201.13287/) (2022)
- [Uncertainty of Joint Neural Contextual Bandit](https://mlsanity.com/paper/2406.02515/) (2024)
- [Bias Mitigation for AI-Feedback Loops in Recommender Systems: A Systematic Literature Review and Taxonomy](https://mlsanity.com/paper/2509.00109/) (2025)
- [Empirical Bayesian Multi-Bandit Learning](https://mlsanity.com/paper/2510.26284/) (2025)
- [Evaluating Deep Vs. Wide & Deep Learners As Contextual Bandits For Personalized Email Promo Recommendations](https://mlsanity.com/paper/2202.00146/) (2022)
- [Interactive Graph Convolutional Filtering](https://mlsanity.com/paper/2309.01453/) (2023)
- [Diversify and Conquer: Bandits and Diversity for an Enhanced E-commerce Homepage Experience](https://mlsanity.com/paper/2309.14046/) (2023)
- [Epsilon non-Greedy: A Bandit Approach for Unbiased Recommendation via Uniform Data](https://mlsanity.com/paper/2310.04855/) (2023)
- [Bayesian Bandit Algorithms with Approximate Inference in Stochastic Linear Bandits](https://mlsanity.com/paper/2406.14071/) (2024)
- [Transforming Location Retrieval at Airbnb: A Journey from Heuristics to Reinforcement Learning](https://mlsanity.com/paper/2408.13399/) (2024)
- [The Bandit's Blind Spot: The Critical Role of User State Representation in Recommender Systems](https://mlsanity.com/paper/2604.26651/) (2026)
- [Distributional Approximate Nearest Neighbour Search for Uncertainty-Aware Retrieval](https://mlsanity.com/paper/2606.04603/) (2026)

## Abstract (arXiv)

Recommender systems trained in a continuous learning fashion are plagued by the feedback loop problem, also known as algorithmic bias. This causes a newly trained model to act greedily and favor items that have already been engaged by users. This behavior is particularly harmful in personalised ads recommendations, as it can also cause new campaigns to remain unexplored. Exploration aims to address this limitation by providing new information about the environment, which encompasses user preference, and can lead to higher long-term reward. In this work, we formulate a display advertising recommender as a contextual bandit and implement exploration techniques that require sampling from the posterior distribution of click-through-rates in a computationally tractable manner. Traditional large-scale deep learning models do not provide uncertainty estimates by default. We approximate these uncertainty measurements of the predictions by employing a bootstrapped model with multiple heads and dropout units. We benchmark a number of different models in an offline simulation environment using a publicly available dataset of user-ads engagements. We test our proposed deep Bayesian bandits algorithm in the offline simulation and online AB setting with large-scale production traffic, where we demonstrate a positive gain of our exploration model.