mlsanityData to 14 Oct 2026

Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain

arXiv 2406.17453 ↗First posted 25 Jun 2024cs.CLcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Questions are essential tools for acquiring the necessary information to complete information-seeking tasks. However, large language models (LLMs), especially open-source models, often perform poorly in generating informative questions, as measured by expected information gain (EIG). In this paper, we propose a method to enhance the informativeness of LLM-generated questions in 20-question game dialogues. We sample multiple questions from the same model (LLAMA 2-CHAT 7B) for each game and create pairs of low-EIG and high-EIG questions to apply a Direct Preference Optimization (DPO) algorithm. Our results show that this method produces more effective questions (in terms of EIG), even in domains different from those used to train the DPO model.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 26 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

11 of 26 references.

References · 11

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  5. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
  6. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 39
  7. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24
  8. 2016Learning to Play Guess Who? and Inventing a Grounded Language as a Consequencecited by 15
  9. 2024Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Samplingcited by 12
  10. 2023Asking More Informative Questions for Grounded Retrievalcited by 6
  11. 2021Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 10.

21 papers.

Sort:

By area:

  1. 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 24
  2. 2025ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMscited by 13
  3. 2025Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like Peoplecited by 9
  4. 2025Post-training for Efficient Communication via Convention Formationcited by 7
  5. 2025The Curious Language Model: Strategic Test-Time Information Acquisitioncited by 5
  6. 2025The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilitiescited by 5
  7. 2025Bridging Information Gaps with Comprehensive Answers: Improving the Diversity and Informativeness of Follow-Up Questionscited by 2
  8. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  9. 2026Probing the Lack of Stable Internal Beliefs in LLMscited by 2
  10. 2026Amortising Bayesian Experimental Design for Sequential Information Gathering in LLMscited by 2
  11. 2025GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Modelscited by 1
  12. 2025Sherlock Your Queries: Learning to Ask the Right Questions for Dialogue-Based Retrievalcited by 1
9 more
  1. 2026Do Reasoning Models Ask Better Questions? A Formal Information-Theoretic Analysis on Multi-Turn LLM Gamescited by 1
  2. 2025Closing the Expression Gap in LLM Instructions via Socratic Questioningcited by 0
  3. 2026Benchmarking Agents in Insurance Underwriting Environmentscited by 0
  4. 2026Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversationcited by 0
  5. 2026Adaptive Querying with AI Persona Priorscited by 0
  6. 2026Ask-E: An Environment for Calibrated Question Generationcited by 0
  7. 2026When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gaincited by 0
  8. 2026CruxBench: A Benchmark of Information Discoverycited by 0
  9. 2026Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Asking Clarifying Questions for Preference Elicitation With Large Language Models
  2. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questions
  3. 2024Improving Socratic Question Generation using Data Augmentation and Preference Optimization
  4. 2025Do LLMs Recognize Your Latent Preferences? A Benchmark for Latent Information Discovery in Personalized Interaction
  5. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models
  6. 2024Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation
  7. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs
  8. 2023Active Preference Inference using Language Models and Probabilistic Reasoning