# ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning

arXiv: https://arxiv.org/abs/2502.14860 · HTML: https://mlsanity.com/paper/2502.14860/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 36 of its 99 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- LLM alignment / RLHF / preference optimisation / [Medical LLM alignment](https://mlsanity.com/alignment/medical-llm-alignment/)

## Builds on (most cited first, 30 of 36)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [HelpSteer2: Open-source dataset for training top-performing reward models](https://mlsanity.com/paper/2406.08673/) (2024)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization](https://mlsanity.com/paper/2310.03708/) (2023)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [Decoding-Time Language Model Alignment with Multiple Objectives](https://mlsanity.com/paper/2406.18853/) (2024)
- [AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models](https://mlsanity.com/paper/2302.07027/) (2023)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework](https://mlsanity.com/paper/2405.11143/) (2024)
- [Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding](https://mlsanity.com/paper/2305.12031/) (2023)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models](https://mlsanity.com/paper/2402.03271/) (2024)
- [A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?](https://mlsanity.com/paper/2409.15277/) (2024)
- [Evaluating Language Models as Synthetic Data Generators](https://mlsanity.com/paper/2412.03679/) (2024)
- [HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversations](https://mlsanity.com/paper/2409.19487/) (2024)
- [MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications](https://mlsanity.com/paper/2409.07314/) (2024)
- [Open (Clinical) LLMs are Sensitive to Instruction Phrasings](https://mlsanity.com/paper/2407.09429/) (2024)
- [Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging](https://mlsanity.com/paper/2502.06876/) (2025)
- [MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback](https://mlsanity.com/paper/2410.13191/) (2024)

## Built on by (most cited first, 25 of 25)

- [Training Proactive and Personalized LLM Agents](https://mlsanity.com/paper/2511.02208/) (2025)
- [PrefPalette: Personalized Preference Modeling with Latent Attributes](https://mlsanity.com/paper/2507.13541/) (2025)
- [TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories](https://mlsanity.com/paper/2511.21322/) (2025)
- [MedKGI: Iterative Differential Diagnosis with Medical Knowledge Graphs and Information-Guided Inquiring](https://mlsanity.com/paper/2512.24181/) (2025)
- [MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaboration](https://mlsanity.com/paper/2601.02702/) (2026)
- [Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration](https://mlsanity.com/paper/2507.23407/) (2025)
- [GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses](https://mlsanity.com/paper/2604.11924/) (2026)
- [Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification](https://mlsanity.com/paper/2601.16400/) (2026)
- [Cold-Start Personalization via Training-Free Priors from Structured World Models](https://mlsanity.com/paper/2602.15012/) (2026)
- [InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning](https://mlsanity.com/paper/2603.05909/) (2026)
- [Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations](https://mlsanity.com/paper/2603.29373/) (2026)
- [Linking Knowledge to Care: Knowledge Graph-Augmented Medical Follow-Up Question Generation](https://mlsanity.com/paper/2603.01252/) (2026)
- [MedConceal: A Benchmark for Clinical Hidden-Concern Reasoning Under Partial Observability](https://mlsanity.com/paper/2604.08788/) (2026)
- [BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum](https://mlsanity.com/paper/2505.21757/) (2025)
- [RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models](https://mlsanity.com/paper/2506.02726/) (2025)
- [Towards Better Health Conversations: The Benefits of Context-seeking](https://mlsanity.com/paper/2510.18880/) (2025)
- [Socratic Students: Teaching Language Models to Learn by Asking Questions](https://mlsanity.com/paper/2512.13102/) (2025)
- [How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting](https://mlsanity.com/paper/2601.11344/) (2026)
- [CA-BED: Conversation-Aware Bayesian Experimental Design](https://mlsanity.com/paper/2606.01182/) (2026)
- [LLM-Guided Evolution for Medical Decision Pipelines](https://mlsanity.com/paper/2606.07342/) (2026)
- [PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents](https://mlsanity.com/paper/2607.25485/) (2026)
- [Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution](https://mlsanity.com/paper/2608.01366/) (2026)
- [Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent](https://mlsanity.com/paper/2608.04772/) (2026)
- [PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue](https://mlsanity.com/paper/2609.27987/) (2026)
- [IsoVRBench: Video Reward Modeling Without Human Feedback](https://mlsanity.com/paper/2610.17234/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often fail to ask effective questions under uncertainty, making them unreliable in domains where proactive information-gathering is essential for decision-making. We present ALignment via Fine-grained Attributes, (ALFA) a framework that improves LLM question-asking by (i) decomposing the notion of a "good" question into a set of theory-grounded attributes (e.g., clarity, relevance), (ii) controllably synthesizing attribute-specific question variations, and (iii) aligning models via preference-based optimization to explicitly learn to ask better questions along these fine-grained attributes. Focusing on clinical reasoning as a case study, we introduce the MediQ-AskDocs dataset, composed of 17k real-world clinical interactions augmented with 80k attribute-specific preference pairs of follow-up questions, as well as a novel expert-annotated interactive healthcare QA task to evaluate question-asking abilities. Models aligned with ALFA reduce diagnostic errors by 56.6% on MediQ-AskDocs compared to SoTA instruction-tuned LLMs, with a question-level win-rate of 64.4% and strong generalizability. Our findings suggest that explicitly guiding question-asking with structured, fine-grained attributes offers a scalable path to improve LLMs, especially in expert application domains.