# Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation

arXiv: https://arxiv.org/abs/2405.00981 · HTML: https://mlsanity.com/paper/2405.00981/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 19 of 19)

- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [MeLU: Meta-Learned User Preference Estimator for Cold-Start Recommendation](https://mlsanity.com/paper/1908.00413/) (2019)
- [Towards Knowledge-Based Recommender Dialog System](https://mlsanity.com/paper/1908.05391/) (2019)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning](https://mlsanity.com/paper/2206.09363/) (2022)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [Interactive Path Reasoning on Graph for Conversational Recommendation](https://mlsanity.com/paper/2007.00194/) (2020)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [Conversational Contextual Bandit: Algorithm and Application](https://mlsanity.com/paper/1906.01219/) (2019)
- [Bayesian Preference Elicitation with Language Models](https://mlsanity.com/paper/2403.05534/) (2024)
- [Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-Start Users](https://mlsanity.com/paper/2005.12979/) (2020)
- [Improving Conversational Recommendation Systems' Quality with Context-Aware Item Meta Information](https://mlsanity.com/paper/2112.08140/) (2021)
- [Gradient-based Optimization for Bayesian Preference Elicitation](https://mlsanity.com/paper/1911.09153/) (2019)
- [Item Recommendation with Variational Autoencoders and Heterogenous Priors](https://mlsanity.com/paper/1807.06651/) (2018)

## Built on by (most cited first, 17 of 17)

- [Retrieval-Augmented Conversational Recommendation with Prompt-based Semi-Structured Natural Language State Tracking](https://mlsanity.com/paper/2406.00033/) (2024)
- [Elaborative Subtopic Query Reformulation for Broad and Indirect Queries in Travel Destination Recommendation](https://mlsanity.com/paper/2410.01598/) (2024)
- [Accelerated Preference Elicitation with LLM-Based Proxies](https://mlsanity.com/paper/2501.14625/) (2025)
- [Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2410.13957/) (2024)
- [Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2505.21907/) (2025)
- [ChemBOMAS: Accelerated BO in Chemistry with LLM-Enhanced Multi-Agent System](https://mlsanity.com/paper/2509.08736/) (2025)
- [Cold-Start Personalization via Training-Free Priors from Structured World Models](https://mlsanity.com/paper/2602.15012/) (2026)
- [Learning Explainable Dense Reward Shapes via Bayesian Optimization](https://mlsanity.com/paper/2504.16272/) (2025)
- [LISTEN to Your Preferences: An LLM Framework for Multi-Objective Selection](https://mlsanity.com/paper/2510.25799/) (2025)
- [Large Language Models for Physics Instrument Design](https://mlsanity.com/paper/2601.07580/) (2026)
- [Flexible Agent Alignment with Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2508.15119/) (2025)
- [AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics](https://mlsanity.com/paper/2608.27818/) (2026)
- [Large Language Model Driven Recommendation](https://mlsanity.com/paper/2408.10946/) (2024)
- [Decision-Centric Design for LLM Systems](https://mlsanity.com/paper/2604.00414/) (2026)
- [Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents](https://mlsanity.com/paper/2604.18122/) (2026)
- [AI YOU Town: Make Friends and Money with Your Digital Twin](https://mlsanity.com/paper/2607.10539/) (2026)
- [In-Context Guidance: Learning Inter-Task Synergies via Numerical Foundational Models for Few-Shot Multitask Optimization](https://mlsanity.com/paper/2609.25836/) (2026)

## Abstract (arXiv)

Designing preference elicitation (PE) methodologies that can quickly ascertain a user's top item preferences in a cold-start setting is a key challenge for building effective and personalized conversational recommendation (ConvRec) systems. While large language models (LLMs) enable fully natural language (NL) PE dialogues, we hypothesize that monolithic LLM NL-PE approaches lack the multi-turn, decision-theoretic reasoning required to effectively balance the exploration and exploitation of user preferences towards an arbitrary item set. In contrast, traditional Bayesian optimization PE methods define theoretically optimal PE strategies, but cannot generate arbitrary NL queries or reason over content in NL item descriptions -- requiring users to express preferences via ratings or comparisons of unfamiliar items. To overcome the limitations of both approaches, we formulate NL-PE in a Bayesian Optimization (BO) framework that seeks to actively elicit NL feedback to identify the best recommendation. Key challenges in generalizing BO to deal with natural language feedback include determining: (a) how to leverage LLMs to model the likelihood of NL preference feedback as a function of item utilities, and (b) how to design an acquisition function for NL BO that can elicit preferences in the infinite space of language. We demonstrate our framework in a novel NL-PE algorithm, PEBOL, which uses: 1) Natural Language Inference (NLI) between user preference utterances and NL item descriptions to maintain Bayesian preference beliefs, and 2) BO strategies such as Thompson Sampling (TS) and Upper Confidence Bound (UCB) to steer LLM query generation. We numerically evaluate our methods in controlled simulations, finding that after 10 turns of dialogue, PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline's MRR@10 of 0.17, despite relying on earlier and smaller LLMs.