# Conversational Contextual Bandit: Algorithm and Application

arXiv: https://arxiv.org/abs/1906.01219 · HTML: https://mlsanity.com/paper/1906.01219/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 30 of 35)

- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [Cost-Effective Online Multi-LLM Selection with Versatile Reward Models](https://mlsanity.com/paper/2405.16587/) (2024)
- [BARCOR: Towards A Unified Framework for Conversational Recommendation Systems](https://mlsanity.com/paper/2203.14257/) (2022)
- [Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation](https://mlsanity.com/paper/2405.00981/) (2024)
- [Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialogue](https://mlsanity.com/paper/2310.14626/) (2023)
- [Comparison-based Conversational Recommender System with Relative Bandit Feedback](https://mlsanity.com/paper/2208.09837/) (2022)
- [Efficient Explorative Key-term Selection Strategies for Conversational Contextual Bandits](https://mlsanity.com/paper/2303.00315/) (2023)
- [Scalable Neural Contextual Bandit for Recommender Systems](https://mlsanity.com/paper/2306.14834/) (2023)
- [Bundle MCR: Towards Conversational Bundle Recommendation](https://mlsanity.com/paper/2207.12628/) (2022)
- [Federated Online Clustering of Bandits](https://mlsanity.com/paper/2208.14865/) (2022)
- [Hierarchical Conversational Preference Elicitation with Bandit Feedback](https://mlsanity.com/paper/2209.06129/) (2022)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Learning to Ask: Conversational Product Search via Representation Learning](https://mlsanity.com/paper/2411.14466/) (2024)
- [Meta Policy Learning for Cold-Start Conversational Recommendation](https://mlsanity.com/paper/2205.11788/) (2022)
- [Lending Interaction Wings to Recommender Systems with Conversational Agents](https://mlsanity.com/paper/2310.04230/) (2023)
- [FedConPE: Efficient Federated Conversational Bandits with Heterogeneous Clients](https://mlsanity.com/paper/2405.02881/) (2024)
- [Conversational Dueling Bandits in Generalized Linear Models](https://mlsanity.com/paper/2407.18488/) (2024)
- [Uncertainty-Aware Instance Reweighting for Off-Policy Learning](https://mlsanity.com/paper/2303.06389/) (2023)
- [Demystifying Online Clustering of Bandits: Enhanced Exploration Under Stochastic and Smoothed Adversarial Contexts](https://mlsanity.com/paper/2501.00891/) (2025)
- [BanditMF: Multi-Armed Bandit Based Matrix Factorization Recommender System](https://mlsanity.com/paper/2106.10898/) (2021)
- [Two-Stage Neural Contextual Bandits for Personalised News Recommendation](https://mlsanity.com/paper/2206.14648/) (2022)
- [Leveraging heterogeneous spillover in maximizing contextual bandit rewards](https://mlsanity.com/paper/2310.10259/) (2023)
- [Leveraging the Power of Conversations: Optimal Key Term Selection in Conversational Contextual Bandits](https://mlsanity.com/paper/2505.21393/) (2025)
- [Reinforced Approximate Exploratory Data Analysis](https://mlsanity.com/paper/2212.06225/) (2022)
- [Contextual Bandit with Herding Effects: Algorithms and Recommendation Applications](https://mlsanity.com/paper/2408.14432/) (2024)
- [Fusing Reward and Dueling Feedback in Stochastic Bandits](https://mlsanity.com/paper/2504.15812/) (2025)
- [Causally-Informed Reinforcement Learning for Adaptive Emotion-Aware Social Media Recommendation](https://mlsanity.com/paper/2511.14768/) (2025)
- [Best Arm Identification in Generalized Linear Bandits via Hybrid Feedback](https://mlsanity.com/paper/2605.05745/) (2026)
- [JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimization](https://mlsanity.com/paper/2606.11425/) (2026)

## Abstract (arXiv)

Contextual bandit algorithms provide principled online learning solutions to balance the exploitation-exploration trade-off in various applications such as recommender systems. However, the learning speed of the traditional contextual bandit algorithms is often slow due to the need for extensive exploration. This poses a critical issue in applications like recommender systems, since users may need to provide feedbacks on a lot of uninterested items. To accelerate the learning speed, we generalize contextual bandit to conversational contextual bandit. Conversational contextual bandit leverages not only behavioral feedbacks on arms (e.g., articles in news recommendation), but also occasional conversational feedbacks on key-terms from the user. Here, a key-term can relate to a subset of arms, for example, a category of articles in news recommendation. We then design the Conversational UCB algorithm (ConUCB) to address two challenges in conversational contextual bandit: (1) which key-terms to select to conduct conversation, (2) how to leverage conversational feedbacks to accelerate the speed of bandit learning. We theoretically prove that ConUCB can achieve a smaller regret upper bound than the traditional contextual bandit algorithm LinUCB, which implies a faster learning speed. Experiments on synthetic data, as well as real datasets from Yelp and Toutiao, demonstrate the efficacy of the ConUCB algorithm.