mlsanityData to 14 Oct 2026

Eliciting Human Preferences with Language Models

arXiv 2310.11589 ↗First posted 17 Oct 2023cs.CL cs.AI cs.LGcited by 92

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) can be directed to perform target tasks by using labeled examples or natural language prompts. But selecting examples or writing prompts for can be challenging--especially in tasks that involve unusual edge cases, demand precise articulation of nebulous preferences, or require an accurate mental model of LM behavior. We propose to use *LMs themselves* to guide the task specification process. In this paper, we introduce **Generative Active Task Elicitation (GATE)**: a learning framework in which models elicit and infer intended behavior through free-form, language-based interaction with users. We study GATE in three domains: email validation, content recommendation, and moral reasoning. In preregistered experiments, we show that LMs prompted to perform GATE (e.g., by generating open-ended questions or synthesizing informative edge cases) elicit responses that are often more informative than user-written prompts or labels. Users report that interactive task elicitation requires less effort than prompting or example labeling and surfaces novel considerations not initially anticipated by users. Our findings suggest that LM-driven elicitation can be a powerful tool for aligning models to complex human preferences and values.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). Builds on 19 of its 55 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

19 of 55 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  7. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  9. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  10. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  11. 2016Cooperative Inverse Reinforcement Learningcited by 300
  12. 2018Probabilistic Model-Agnostic Meta-Learningcited by 256
7 more
  1. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  2. 2017Generative Adversarial Active Learningcited by 70
  3. 2020Robustness to Spurious Correlations via Human Annotationscited by 62
  4. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 43
  5. 2022Active Learning Helps Pretrained Models Learn the Intended Taskcited by 33
  6. 2022Task Ambiguity in Humans and Language Modelscited by 28
  7. 2020Robust Active Preference Elicitationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 29; 2025: 25; 2026: 34.

92 papers.

Sort:

By area:

  1. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  2. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 124
  3. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  4. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  5. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  6. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 76
  7. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  8. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  9. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  10. 2023Grounding Gaps in Language Model Generationscited by 47
  11. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  12. 2024Bayesian Preference Elicitation with Language Modelscited by 34
38 more
  1. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  2. 2023Active Preference Inference using Language Models and Probabilistic Reasoningcited by 29
  3. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 29
  4. 2024Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoningcited by 24
  5. 2024ChatShop: Interactive Information Seeking with Language Agentscited by 24
  6. 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 24
  7. 2024A Survey of Conversational Searchcited by 22
  8. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 22
  9. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 19
  10. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 19
  11. 2024Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitationcited by 17
  12. 2024On Subjective Uncertainty Quantification and Calibration in Natural Language Generationcited by 16
  13. 2024ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agentscited by 15
  14. 2024Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agentcited by 14
  15. 2025Thinking beyond the anthropomorphic paradigm benefits LLM researchcited by 14
  16. 2024Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Samplingcited by 12
  17. 2024Toward In-Context Teaching: Adapting Examples to Students' Misconceptionscited by 12
  18. 2025Guiding Audio Editing with Audio Language Modelcited by 12
  19. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  20. 2026Value of Information: A Framework for Human-Agent Communicationcited by 12
  21. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  22. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  23. 2025Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like Peoplecited by 9
  24. 2024Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Inputcited by 8
  25. 2025PrefDisco: Benchmarking Proactive Personalized Reasoningcited by 7
  26. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 7
  27. 2026MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaborationcited by 7
  28. 2024INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Modelscited by 6
  29. 2025Substance over Style: Evaluating Proactive Conversational Coaching Agentscited by 6
  30. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  31. 2025Completion neq Collaboration: Scaling Collaborative Effort with Agentscited by 6
  32. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 6
  33. 2026Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviorscited by 6
  34. 2024Towards Unified Alignment Between Agents, Humans, and Environmentcited by 5
  35. 2025TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preferencecited by 5
  36. 2025Referential ambiguity and clarification requests: comparing human and LLM behaviourcited by 5
  37. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  38. 2026From Words to Widgets for Controllable LLM Generationcited by 5

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Context-Aware Preference Modeling for Language Models
  2. 2026When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
  3. 2024A Survey on Human Preference Learning for Large Language Models
  4. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  5. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences
  6. 2024Aligning language models with human preferences
  7. 2023Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responses
  8. 2026Language Model Goal Selection Differs from Humans' in a Self-Directed Learning Task