# Hierarchical Conversational Preference Elicitation with Bandit Feedback

arXiv: https://arxiv.org/abs/2209.06129 · HTML: https://mlsanity.com/paper/2209.06129/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 6 Sep 2022.

## Builds on (most cited first, 20 of 20)

- [Towards Deep Conversational Recommendations](https://mlsanity.com/paper/1812.07617/) (2018)
- [Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion](https://mlsanity.com/paper/2007.04032/) (2020)
- [Conversational Recommender System](https://mlsanity.com/paper/1806.03277/) (2018)
- [Towards Knowledge-Based Recommender Dialog System](https://mlsanity.com/paper/1908.05391/) (2019)
- [Towards Conversational Recommendation over Multi-Type Dialogs](https://mlsanity.com/paper/2005.03954/) (2020)
- [Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems](https://mlsanity.com/paper/2002.09102/) (2020)
- [Towards Topic-Guided Conversational Recommender System](https://mlsanity.com/paper/2010.04125/) (2020)
- [Interactive Path Reasoning on Graph for Conversational Recommendation](https://mlsanity.com/paper/2007.00194/) (2020)
- [Neural Contextual Bandits with Deep Representation and Shallow Exploration](https://mlsanity.com/paper/2012.01780/) (2020)
- [Meta-Thompson Sampling](https://mlsanity.com/paper/2102.06129/) (2021)
- [Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning](https://mlsanity.com/paper/2105.09710/) (2021)
- [Conversational Contextual Bandit: Algorithm and Application](https://mlsanity.com/paper/1906.01219/) (2019)
- [Hierarchical Bayesian Bandits](https://mlsanity.com/paper/2111.06929/) (2021)
- [Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Users](https://mlsanity.com/paper/2005.12979/) (2020)
- [No Regrets for Learning the Prior in Bandits](https://mlsanity.com/paper/2107.06196/) (2021)
- [Towards Question-based Recommender Systems](https://mlsanity.com/paper/2005.14255/) (2020)
- [Metadata-based Multi-Task Bandits with Bayesian Hierarchical Models](https://mlsanity.com/paper/2108.06422/) (2021)
- [Deep Hierarchy in Bandits](https://mlsanity.com/paper/2202.01454/) (2022)
- [Comparison-based Conversational Recommender System with Relative Bandit Feedback](https://mlsanity.com/paper/2208.09837/) (2022)
- [Categorized Bandits](https://mlsanity.com/paper/2005.01656/) (2020)

## Built on by (most cited first, 10 of 10)

- [On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots](https://mlsanity.com/paper/2406.01633/) (2024)
- [Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution](https://mlsanity.com/paper/2506.17670/) (2025)
- [FedConPE: Efficient Federated Conversational Bandits with Heterogeneous Clients](https://mlsanity.com/paper/2405.02881/) (2024)
- [HR-Bandit: Human-AI Collaborated Linear Recourse Bandit](https://mlsanity.com/paper/2410.14640/) (2024)
- [Leveraging the Power of Conversations: Optimal Key Term Selection in Conversational Contextual Bandits](https://mlsanity.com/paper/2505.21393/) (2025)
- [Fair Algorithms with Probing for Multi-Agent Multi-Armed Bandits](https://mlsanity.com/paper/2506.14988/) (2025)
- [Trading Vector Data in Vector Databases](https://mlsanity.com/paper/2511.07139/) (2025)
- [Hierarchical Dataset Selection for High-Quality Data Sharing](https://mlsanity.com/paper/2512.10952/) (2025)
- [LIBRA: Language Model Informed Bandit Recourse Algorithm for Personalized Treatment Planning](https://mlsanity.com/paper/2601.11905/) (2026)
- [RPS: Information Elicitation with Reinforcement Prompt Selection](https://mlsanity.com/paper/2604.13817/) (2026)

## Abstract (arXiv)

The recent advances of conversational recommendations provide a promising way to efficiently elicit users' preferences via conversational interactions. To achieve this, the recommender system conducts conversations with users, asking their preferences for different items or item categories. Most existing conversational recommender systems for cold-start users utilize a multi-armed bandit framework to learn users' preference in an online manner. However, they rely on a pre-defined conversation frequency for asking about item categories instead of individual items, which may incur excessive conversational interactions that hurt user experience. To enable more flexible questioning about key-terms, we formulate a new conversational bandit problem that allows the recommender system to choose either a key-term or an item to recommend at each round and explicitly models the rewards of these actions. This motivates us to handle a new exploration-exploitation (EE) trade-off between key-term asking and item recommendation, which requires us to accurately model the relationship between key-term and item rewards. We conduct a survey and analyze a real-world dataset to find that, unlike assumptions made in prior works, key-term rewards are mainly affected by rewards of representative items. We propose two bandit algorithms, Hier-UCB and Hier-LinUCB, that leverage this observed relationship and the hierarchical structure between key-terms and items to efficiently learn which items to recommend. We theoretically prove that our algorithm can reduce the regret bound's dependency on the total number of items from previous work. We validate our proposed algorithms and regret bound on both synthetic and real-world data.