# Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search

arXiv: https://arxiv.org/abs/2008.03717 · HTML: https://mlsanity.com/paper/2008.03717/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [Zero-shot Clarifying Question Generation for Conversational Search](https://mlsanity.com/paper/2301.12660/) (2023)
- [Analysing Mixed Initiatives and Search Strategies during Conversational Search](https://mlsanity.com/paper/2109.05955/) (2021)
- [Context-Aware Target Apps Selection and Recommendation for Enhancing Personal Mobile Assistants](https://mlsanity.com/paper/2101.03394/) (2021)
- [Learning to Ask: Conversational Product Search via Representation Learning](https://mlsanity.com/paper/2411.14466/) (2024)
- [Conversational Search with Mixed-Initiative -- Asking Good Clarification Questions backed-up by Passage Retrieval](https://mlsanity.com/paper/2112.07308/) (2021)
- [Query Understanding in LLM-based Conversational Information Seeking](https://mlsanity.com/paper/2504.06356/) (2025)
- [Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search](https://mlsanity.com/paper/2401.11463/) (2024)
- [Corpus-informed Retrieval Augmented Generation of Clarifying Questions](https://mlsanity.com/paper/2409.18575/) (2024)
- [Keyword Extraction for Improved Document Retrieval in Conversational Search](https://mlsanity.com/paper/2109.05979/) (2021)
- [Leveraging Knowledge Graph Embedding for Effective Conversational Recommendation](https://mlsanity.com/paper/2408.01342/) (2024)
- [Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search](https://mlsanity.com/paper/2602.08700/) (2026)

## Abstract (arXiv)

Recent research on conversational search highlights the importance of mixed-initiative in conversations. To enable mixed-initiative, the system should be able to ask clarifying questions to the user. However, the ability of the underlying ranking models (which support conversational search) to account for these clarifying questions and answers has not been analysed when ranking documents, at large. To this end, we analyse the performance of a lexical ranking model on a conversational search dataset with clarifying questions. We investigate, both quantitatively and qualitatively, how different aspects of clarifying questions and user answers affect the quality of ranking. We argue that there needs to be some fine-grained treatment of the entire conversational round of clarification, based on the explicit feedback which is present in such mixed-initiative settings. Informed by our findings, we introduce a simple heuristic-based lexical baseline, that significantly outperforms the existing naive baselines. Our work aims to enhance our understanding of the challenges present in this particular task and inform the design of more appropriate conversational ranking models.