# Eliciting Human Preferences with Language Models

arXiv: https://arxiv.org/abs/2310.11589 · HTML: https://mlsanity.com/paper/2310.11589/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 55 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Probabilistic Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.02817/) (2018)
- [Understanding the Failure Modes of Out-of-Distribution Generalization](https://mlsanity.com/paper/2010.15775/) (2020)
- [Generative Adversarial Active Learning](https://mlsanity.com/paper/1702.07956/) (2017)
- [Robustness to Spurious Correlations via Human Annotations](https://mlsanity.com/paper/2007.06661/) (2020)
- [Hidden Incentives for Auto-Induced Distributional Shift](https://mlsanity.com/paper/2009.09153/) (2020)
- [Active Learning Helps Pretrained Models Learn the Intended Task](https://mlsanity.com/paper/2204.08491/) (2022)
- [Task Ambiguity in Humans and Language Models](https://mlsanity.com/paper/2212.10711/) (2022)
- [Robust Active Preference Elicitation](https://mlsanity.com/paper/2003.01899/) (2020)

## Built on by (most cited first, 30 of 92)

- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Evaluating and Mitigating Discrimination in Language Model Decisions](https://mlsanity.com/paper/2312.03689/) (2023)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [PERSONA: A Reproducible Testbed for Pluralistic Alignment](https://mlsanity.com/paper/2407.17387/) (2024)
- [Aligning LLMs with Individual Preferences via Interaction](https://mlsanity.com/paper/2410.03642/) (2024)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [Bayesian Preference Elicitation with Language Models](https://mlsanity.com/paper/2403.05534/) (2024)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Active Preference Inference using Language Models and Probabilistic Reasoning](https://mlsanity.com/paper/2312.12009/) (2023)
- [Navigating Rifts in Human-LLM Grounding: Study and Benchmark](https://mlsanity.com/paper/2503.13975/) (2025)
- [Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning](https://mlsanity.com/paper/2402.06025/) (2024)
- [ChatShop: Interactive Information Seeking with Language Agents](https://mlsanity.com/paper/2404.09911/) (2024)
- [BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design](https://mlsanity.com/paper/2508.21184/) (2025)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions](https://mlsanity.com/paper/2508.01674/) (2025)
- [Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas](https://mlsanity.com/paper/2501.11549/) (2025)
- [Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents](https://mlsanity.com/paper/2603.26233/) (2026)
- [Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation](https://mlsanity.com/paper/2405.00981/) (2024)
- [On Subjective Uncertainty Quantification and Calibration in Natural Language Generation](https://mlsanity.com/paper/2406.05213/) (2024)
- [ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents](https://mlsanity.com/paper/2411.00927/) (2024)
- [Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent](https://mlsanity.com/paper/2411.01344/) (2024)
- [Thinking beyond the anthropomorphic paradigm benefits LLM research](https://mlsanity.com/paper/2502.09192/) (2025)
- [Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling](https://mlsanity.com/paper/2402.19471/) (2024)
- [Toward In-Context Teaching: Adapting Examples to Students' Misconceptions](https://mlsanity.com/paper/2405.04495/) (2024)
- [Guiding Audio Editing with Audio Language Model](https://mlsanity.com/paper/2509.21625/) (2025)

## Abstract (arXiv)

Language models (LMs) can be directed to perform target tasks by using labeled examples or natural language prompts. But selecting examples or writing prompts for can be challenging--especially in tasks that involve unusual edge cases, demand precise articulation of nebulous preferences, or require an accurate mental model of LM behavior. We propose to use *LMs themselves* to guide the task specification process. In this paper, we introduce **Generative Active Task Elicitation (GATE)**: a learning framework in which models elicit and infer intended behavior through free-form, language-based interaction with users. We study GATE in three domains: email validation, content recommendation, and moral reasoning. In preregistered experiments, we show that LMs prompted to perform GATE (e.g., by generating open-ended questions or synthesizing informative edge cases) elicit responses that are often more informative than user-written prompts or labels. Users report that interactive task elicitation requires less effort than prompting or example labeling and surfaces novel considerations not initially anticipated by users. Our findings suggest that LM-driven elicitation can be a powerful tool for aligning models to complex human preferences and values.