# Bridging Information Gaps with Comprehensive Answers: Improving the Diversity and Informativeness of Follow-Up Questions

arXiv: https://arxiv.org/abs/2502.17715 · HTML: https://mlsanity.com/paper/2502.17715/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 18 of 18)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [Zero-shot Fact Verification by Claim Generation](https://mlsanity.com/paper/2105.14682/) (2021)
- [Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge](https://mlsanity.com/paper/2104.06828/) (2021)
- [Semantic Graphs for Generating Deep Questions](https://mlsanity.com/paper/2004.12704/) (2020)
- [Inquisitive Question Generation for High Level Text Comprehension](https://mlsanity.com/paper/2010.01657/) (2020)
- [Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts](https://mlsanity.com/paper/2205.13884/) (2022)
- [Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain](https://mlsanity.com/paper/2406.17453/) (2024)
- [Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask](https://mlsanity.com/paper/2204.02908/) (2022)
- [Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations](https://mlsanity.com/paper/2004.14530/) (2020)
- [What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys](https://mlsanity.com/paper/2205.10977/) (2022)
- [Learning to Retrieve Iteratively for In-Context Learning](https://mlsanity.com/paper/2406.14739/) (2024)
- [Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation](https://mlsanity.com/paper/2205.01730/) (2022)
- [FOLLOWUPQG: Towards Information-Seeking Follow-up Question Generation](https://mlsanity.com/paper/2309.05007/) (2023)

## Built on by (most cited first, 2 of 2)

- [InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning](https://mlsanity.com/paper/2603.05909/) (2026)
- [Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data](https://mlsanity.com/paper/2603.00059/) (2026)

## Abstract (arXiv)

Generating diverse follow-up questions that uncover missing information remains challenging for conversational agents, particularly when they run on small, locally hosted models. To address this, we develop an information-gap-driven knowledge distillation pipeline in which a teacher LLM generates a comprehensive answer, contrasts it with the initial answer to identify information gaps, and formulates gap-bridging follow-up questions. Using this pipeline, we augment the existing FollowupQG dataset tenfold. We then fine-tune smaller student models on the augmented dataset to distill the teacher's knowledge. Experiments with selected teacher-student model pairs show that fine-tuned students achieve significantly higher informativeness and diversity than variations trained on the original dataset. These findings indicate that our pipeline, which mirrors the human cognitive process of information seeking, provides an efficient distillation channel from state-of-the-art LLMs to smaller models, enabling resource-constrained conversational systems to generate more diverse and informative follow-up questions.