# Alexpaca: Learning Factual Clarification Question Generation Without Examples

arXiv: https://arxiv.org/abs/2310.11571 · HTML: https://mlsanity.com/paper/2310.11571/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information](https://mlsanity.com/paper/1805.04655/) (2018)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots](https://mlsanity.com/paper/1801.01957/) (2018)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance](https://mlsanity.com/paper/2210.08817/) (2022)
- [Answer-based Adversarial Training for Generating Clarification Questions](https://mlsanity.com/paper/1904.02281/) (2019)
- [ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations](https://mlsanity.com/paper/2212.10409/) (2022)
- [A Survey on Asking Clarification Questions Datasets in Conversational Systems](https://mlsanity.com/paper/2305.15933/) (2023)
- [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://mlsanity.com/paper/2305.13808/) (2023)
- [Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations](https://mlsanity.com/paper/2004.14530/) (2020)
- [Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systems](https://mlsanity.com/paper/2111.13463/) (2021)
- [Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue](https://mlsanity.com/paper/2106.15550/) (2021)

## Built on by (most cited first, 2 of 2)

- [Can Code Language Models Learn Clarification-Seeking Behaviors?](https://mlsanity.com/paper/2504.16331/) (2025)
- [Program Synthesis Dialog Agents for Interactive Decision-Making](https://mlsanity.com/paper/2502.19610/) (2025)

## Abstract (arXiv)

Real-life tasks such as giving legal or technical advice often lack complete context at the outset and can have disparate answers depending thereon. The ability to derive missing factual information by asking clarifying questions (ACQ) is an important element of real-life collaboration on such reasoning tasks. Existing factual clarification question challenges evaluate generations based on word overlap or human evaluations. Recent work explores generating a response to the clarifying question then evaluating its utility directly. So far, these tasks are limited to disambiguating the user's intent rather than concrete facts about the situation. The factual domain presents unique challenges since responses to clarification questions must be factually true for accurate evaluation. To enable evaluation of factual domain clarification question generation, We present a new task that focuses on the ability to elicit missing information in multi-hop reasoning tasks. The task, HotpotQA-FLM, can be evaluated automatically, making it convenient for benchmarking language models. We observe that humans outperform GPT-4 by a large margin, while Llama 3 8B Instruct does not even beat the dummy baseline in some metrics. Finally, we find by fine-tuning Llama 3 8B Instruct on its own generations, filtered via rejection sampling, we can improve information recovery by 27.6 percent.