mlsanityData to 14 Oct 2026

Bayesian Preference Elicitation with Language Models

arXiv 2403.05534 ↗First posted 8 Mar 2024cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Aligning AI systems to users' interests requires understanding and incorporating humans' complex values and preferences. Recently, language models (LMs) have been used to gather information about the preferences of human users. This preference data can be used to fine-tune or guide other LMs and/or AI systems. However, LMs have been shown to struggle with crucial aspects of preference learning: quantifying uncertainty, modeling human mental states, and asking informative questions. These challenges have been addressed in other areas of machine learning, such as Bayesian Optimal Experimental Design (BOED), which focus on designing informative queries within a well-defined feature space. But these methods, in turn, are difficult to scale and apply to real-world problems where simply identifying the relevant features can be difficult. We introduce OPEN (Optimal Preference Elicitation with Natural language) a framework that uses BOED to guide the choice of informative questions and an LM to extract features and translate abstract BOED queries into natural language questions. By combining the flexibility of LMs with the rigor of BOED, OPEN can optimize the informativity of queries while remaining adaptable to real-world domains. In user studies, we find that OPEN outperforms existing LM- and BOED-based methods for preference elicitation.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 15 of its 47 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

15 of 47 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2017Deep Bayesian Active Learning with Image Datacited by 819
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  8. 2019Variational Bayesian Optimal Experimental Designcited by 116
  9. 2023Eliciting Human Preferences with Language Modelscited by 92
  10. 2022Inferring Rewards from Language in Contextcited by 50
  11. 2019Human few-shot learning of compositional instructionscited by 47
  12. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
3 more
  1. 2023Active Preference Inference using Language Models and Probabilistic Reasoningcited by 29
  2. 2022Interactively Learning Preference Constraints in Linear Banditscited by 11
  3. 2020Robust Active Preference Elicitationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 17.

34 papers.

Sort:

By area:

  1. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  2. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  3. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  4. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  5. 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 24
  6. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 19
  7. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 19
  8. 2024Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitationcited by 17
  9. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 17
  10. 2026On the Reliability of Computer Use Agentscited by 10
  11. 2025Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like Peoplecited by 9
  12. 2025Adaptive Elicitation of Latent Information Using Natural Languagecited by 7
22 more
  1. 2024INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Modelscited by 6
  2. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 6
  3. 2025TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preferencecited by 5
  4. 2024Goal Inference from Open-Ended Dialogcited by 4
  5. 2026MedClarify: An information-seeking AI agent for medical diagnosis with case-specific follow-up questionscited by 4
  6. 2024Comparison-based Active Preference Learning for Multi-dimensional Personalizationcited by 3
  7. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  8. 2026Cold-Start Personalization via Training-Free Priors from Structured World Modelscited by 3
  9. 2025Flexible Agent Alignment with Goal Inference from Open-Ended Dialogcited by 1
  10. 2026Game of Thought: Robust Information Seeking with Large Language Models Using Game Theorycited by 1
  11. 2026Visual Persuasion: What Influences Decisions of Vision-Language Models?cited by 1
  12. 2026What Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviewscited by 1
  13. 2026Response-Aware User Memory Selection for LLM Personalizationcited by 1
  14. 2026Embeddings for Preferences, Not Semanticscited by 1
  15. 2026Interactive Task Alignment as a POMDPcited by 1
  16. 2026An AI Scientist that Doesn't Drift: Taste, Structure, and Falsifiable Findings in a Quadruped Navigation Research Loopcited by 1
  17. 2024Large Language Model Driven Recommendationcited by 0
  18. 2026Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documentscited by 0
  19. 2026Adaptive Querying with AI Persona Priorscited by 0
  20. 2026CA-BED: Conversation-Aware Bayesian Experimental Designcited by 0
  21. 2026Preference Reasoning under Indeterminacy in Large Language Modelscited by 0
  22. 2026Rational Clarification by Assistive Agents via Value-of-Information Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Models
  2. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Language
  3. 2024A Survey on Human Preference Learning for Large Language Models
  4. 2024Aligning language models with human preferences
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2024Optimizing Language Models for Human Preferences is a Causal Inference Problem
  7. 2024Improving Context-Aware Preference Modeling for Language Models
  8. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation