# Context Uncertainty in Contextual Bandits with Applications to Recommender Systems

arXiv: https://arxiv.org/abs/2202.00805 · HTML: https://mlsanity.com/paper/2202.00805/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 25 of 25)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Session-based Recommendation with Graph Neural Networks](https://mlsanity.com/paper/1811.00855/) (2018)
- [Neural Attentive Session-based Recommendation](https://mlsanity.com/paper/1711.04725/) (2017)
- [Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits](https://mlsanity.com/paper/1402.0555/) (2014)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Generalization and Exploration via Randomized Value Functions](https://mlsanity.com/paper/1402.0635/) (2014)
- [Collaborative Deep Learning for Recommender Systems](https://mlsanity.com/paper/1409.2944/) (2014)
- [Practical Contextual Bandits with Regression Oracles](https://mlsanity.com/paper/1803.01088/) (2018)
- [Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks](https://mlsanity.com/paper/1706.04148/) (2017)
- [Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning](https://mlsanity.com/paper/1805.10000/) (2018)
- [Towards Bayesian Deep Learning: A Framework and Some Existing Methods](https://mlsanity.com/paper/1608.06884/) (2016)
- [Bilinear Bandits with Low-rank Structure](https://mlsanity.com/paper/1901.02470/) (2019)
- [Distributed Clustering of Linear Bandits in Peer to Peer Networks](https://mlsanity.com/paper/1604.07706/) (2016)
- [Zero-Shot Recommender Systems](https://mlsanity.com/paper/2105.08318/) (2021)
- [Seq2Slate: Re-ranking and Slate Optimization with RNNs](https://mlsanity.com/paper/1810.02019/) (2018)
- [Natural-Parameter Networks: A Class of Probabilistic Neural Networks](https://mlsanity.com/paper/1611.00448/) (2016)
- [Stochastic Low-Rank Bandits](https://mlsanity.com/paper/1712.04644/) (2017)
- [Fast Distributed Bandits for Online Recommendation Systems](https://mlsanity.com/paper/2007.08061/) (2020)
- [Correcting Exposure Bias for Link Recommendation](https://mlsanity.com/paper/2106.07041/) (2021)
- [Towards Neural Mixture Recommender for Long Range Dependent User Sequences](https://mlsanity.com/paper/1902.08588/) (2019)
- [Collaborative Recurrent Autoencoder: Recommend while Learning to Fill in the Blanks](https://mlsanity.com/paper/1611.00454/) (2016)
- [Neural Contextual Bandits with UCB-based Exploration](https://mlsanity.com/paper/1911.04462/) (2019)
- [Quantifying Long Range Dependence in Language and User Behavior to improve RNNs](https://mlsanity.com/paper/1905.09414/) (2019)

## Built on by (most cited first, 4 of 4)

- [Two-Stage Neural Contextual Bandits for Personalised News Recommendation](https://mlsanity.com/paper/2206.14648/) (2022)
- [Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing](https://mlsanity.com/paper/2306.06599/) (2023)
- [Towards Domain Adaptive Neural Contextual Bandits](https://mlsanity.com/paper/2406.09564/) (2024)
- [Safety by Design: Realized-Cost Constraints for Contextual Bandits with Continuous Actions](https://mlsanity.com/paper/2608.26755/) (2026)

## Abstract (arXiv)

Recurrent neural networks have proven effective in modeling sequential user feedbacks for recommender systems. However, they usually focus solely on item relevance and fail to effectively explore diverse items for users, therefore harming the system performance in the long run. To address this problem, we propose a new type of recurrent neural networks, dubbed recurrent exploration networks (REN), to jointly perform representation learning and effective exploration in the latent space. REN tries to balance relevance and exploration while taking into account the uncertainty in the representations. Our theoretical analysis shows that REN can preserve the rate-optimal sublinear regret even when there exists uncertainty in the learned representations. Our empirical study demonstrates that REN can achieve satisfactory long-term rewards on both synthetic and real-world recommendation datasets, outperforming state-of-the-art models.